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TEACHERS' RESPONSES TO HIGH-STAKES TESTING 
AND THE VALIDITY OF GAINS: A PILOT STUDY 



Daniel M. Koretz 

CRESST/Harvard Graduate School of Education 

Laura S. Hamilton 
CRESST/RAND Education 

Abstract 

Previous studies of the validity of gains on high-stakes tests have compared trends in 
scores on a high-stakes test to trends on a lower-stakes test, such as NAEP. However, 
generalizability of gains is likely to be incomplete even when gains are meaningful 
because of differences in the inferences the two tests are designed to support. Therefore, 
this simple approach is useful only when the disparity in trends on the two tests is very 
large. A more sensitive but difficult approach requires identifying the specific aspects of 
performance that increase by varying amounts and comparing these to the specific 
inferences users base on the score increases. A key to this approach may be identifying 
the aspects of performance that teachers focus on in their attempts to raise scores. This 
report presents the results of a pilot study evaluating several types of survey questions 
designed to elicit from teachers detailed information on their instructional responses to 
testing. The types of responses explored are those that previous CRESST work (Koretz, 
McCaffrey, & Hamilton, 2001) suggested are important for validating score gains. Of the 
formats used, the most promising appears to be questions, the prompts for which are 
actual test items, including both items from the high-stakes test for which the teachers 
are preparing and other tests. 



Background 

Advocates of test-based accountability argue that it improves student 
achievement by helping teachers focus on important content, providing incentives 
for good teaching, and producing information that can be used to make decisions 
about students, teachers, and schools. For such systems to work as intended, the 
policies must promote good instruction, and any resulting increases in test scores 
must support valid inferences about increased student achievement. Existing 
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evidence about the effects of high-stakes testing is both incomplete and inconsistent, 
but it indicates that there are reasons to question both of these assumptions. 
Research has shown that high-stakes testing does indeed influence instruction, but 
these effects are complex and comprise both desirable and undesirable changes in 
practice. For example, teachers in districts or states where high stakes are associated 
with test results tend to focus on tested material and de-emphasize untested 
material (see Stecher, 2002, for a recent review). Similarly, research indicates that the 
gains in scores on high-stakes tests often generalize poorly (or not at all) to other 
tests of the same domain, raising doubts about the extent to which these gains 
provide valid evidence of improved student performance (Klein, Hamilton, 
McCaffrey, & Stecher, 2000; Koretz & Barron, 1998; Koretz, Linn, Dunbar, & 
Shepard, 1991; Linn, 2000). 

Validity is a property of inferences made on the basis of test scores, rather than 
of the tests themselves (Messick, 1989), and therefore the validity of these gains 
depends on the specific inferences users based on them. For example, if parents 
believe that improved math scores at their local high school represent broad 
improvements in mathematical proficiency, then a failure of generalization of gains 
to other tests that support similar inferences — say, the SAT, ACT, or the National 
Assessment of Educational Progress (NAEP) — undermines inferences about 
improved achievement. In contrast, if their inferences are focused on the district or 
state content standards and users are not interested in aspects of mathematics not 
captured in those standards, information about the generalizability of gains to other 
tests may be less relevant. 

Research to date has been unable to distinguish with any precision meaningful 
gains in scores from score inflation. In some cases, the failure of generalizability is so 
great as to leave little doubt that gains have been inflated (e.g., Klein et al., 2000; 
Koretz & Barron, 1998; Koretz et al., 1991), but even then, it has not always been 
feasible to differentiate clearly between the portion of gains that is meaningful and 
the portion that is not. For example, Koretz and Barron found that gains in 
mathematics on Kentucky's KIRIS test were roughly four times as large as the gains 
Kentucky students showed on NAEP, even though the KIRIS framework was 
explicitly modeled after the NAEP framework. In the view of the authors, this 
clearly indicated that gains on KIRIS were substantially inflated, but it did not 
necessarily indicate that three fourths of the score gains could be construed as 
inflation. 
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This report describes a pilot study testing the use of new types of survey 
questions to elicit from teachers detailed information about responses to testing 
needed to better evaluate gains in scores. It is one of a series of efforts by CRESST to 
help develop new approaches better able to distinguish score inflation from 
meaningful gains in scores. 

The Link Between Teachers' Behaviors and the Validity of Gains 

The validity of score gains on high-stakes tests is inextricably intertwined with 
the instructional responses of teachers to the testing program. To the extent that 
teachers respond with approaches that bolster students' mastery of the domains 
about which users draw inferences, increases in scores will warrant inferences about 
improved student achievement. On the other hand, a wide variety of teacher 
behaviors have the potential to increase scores without similarly increasing the 
achievement that the scores are intended to represent. Koretz et al. (2001) suggest 
that in terms of their effects on the validity of gains, teachers' responses to testing 
can be grouped into seven categories of test preparation: 

1) teaching more 

2) working harder 

3) working more effectively 

4) reallocation 

5) alignment 

6) coaching 

7) cheating 

The first three of these responses are likely to produce unambiguously 
meaningful gains in scores. For example, if teachers do not exceed the limit of 
students to profit from additional work, teaching more and working harder will 
improve achievement, and working more effectively by definition will increase 
achievement. At the other extreme, cheating can never produce meaningful gains in 
achievement. Our concern therefore is with the three remaining types of behavioral 
response: reallocation, alignment, and coaching. 

Reallocation refers to shifts in instructional resources - e.g., teachers' time or 
students' study time - among various elements of student performance, in order to 
focus these resources more closely on the specific content of the test used for high 
stakes. Koretz et al. (2001) note that each relevant element of performance is given 
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some degree of emphasis (which could be no emphasis at all) by the characteristics 
of the test. The aspects of the test that can contribute to this emphasis are diverse. 
They include not only the intended content, but also unintended content or 
cognitive demands and the style of items or tasks into which the content is 
embedded. For example, they note a secondary mathematics test that gave 
considerable emphasis to coordinate geometry. They surmised that this emphasis 
was inadvertent, for the standards the test was designed to measure did not mention 
coordinate geometry. This emphasis may have arisen because coordinate geometry 
provides a useful way to assess many aspects of elementary algebra, which was 
emphasized by the standards. Koretz et al. argue that to evaluate the validity of 
gains, it is necessary to formalize this notion of emphasis. They suggest labeling the 
emphasis accorded to a given element of performance its "effective test weight." 
They use the term "effective" because the emphasis may not be by design, and they 
define this weight as the influence of changes in performance on that element on 
total scores. (Specifically, it is the partial derivative of scores with respect to changes 
in performance on that element.) Reallocation occurs when teachers or others believe 
they have discerned the effective weights in the high-stakes test and shift their 
instructional resources to match these weights more closely. Reallocation is 
commonly found in studies of teachers' responses to high stakes testing (e.g., Koretz 
& Barron, 1998; Koretz, Barron, Mitchell, & Stecher, 1996; Koretz, Mitchell, Barron, & 
Keith, 1996; Stecher, 2002). 

The mix of meaningful and inflated gains produced by reallocation depends on 
several factors: the material receiving greater emphasis, the material receiving 
reduced emphasis, and the importance of both to the specific inferences drawn by 
users of the test scores (Koretz et al., 2001). The inferences drawn by users also 
reflect a set of weights, albeit usually tacit and poorly formed. For example, a user of 
scores from a ninth-grade mathematics test may place a large weight on basic 
algebra, in effect assuming that a substantial part of an increase in scores reflects 
improved mastery of algebra. To the extent that teachers reallocate resources to 
elements that receive high weights in users' inferences, the validity of gains is 
enhanced; to the extent that they reduce resources to elements to which users give 
substantial weights, the validity of gains is undermined. 

Alignment between standards (or curricula) and tests is often presented as an 
unmitigated good, but from the perspective of the validity of score gains, it is 
nothing more than a specialized case of reallocation in which the material receiving 
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additional emphasis is consistent with standards (i.e., in which the material given 
additional emphasis in instruction was considered important by those setting the 
standards and was therefore emphasized in them). For several reasons, however, 
alignment is not sufficient to ensure validity of gains. First, it is not always certain 
that the standards as written convey accurately and completely the content and 
s kill s intended by those establishing the standards. Second, the validity of gains 
hinges not primarily on the inferences intended by those setting the standards and 
designing the test, but more importantly on the actual inferences drawn by users of 
the scores. Users' inferences may differ substantially from those of the standard 
setters. Third, and most important, validity depends on importance of both the 
material receiving more emphasis and the material receiving less emphasis to the 
inferences users base on test scores. Flence, alignment, while increasing the 
probability that students will learn material deemed by the standards-setters to be 
important, offers no guarantee against score inflation. 

Coaching is the most difficult of these categories of test preparation to describe, 
and it shades into both reallocation at one extreme and cheating at the other. 
Coaching refers to a focusing of instructional resources on narrow, specific aspects 
of the particular test. These may be either substantive or non-substantive, and the 
aspects receiving greater emphasis may or may not be those intentionally 
emphasized by the developers of the test. For example, a teacher may recognize that 
items in a given test always present regular polygons and therefore may focus 
instruction only on those, at the expense of irregular polygons. Alternatively, the 
teacher may recognize that items of a given type are usually presented or scored in a 
particular way and may focus on these details rather than on the content more 
broadly. To the extent that the inferences drawn by users are not limited to these 
specific aspects of the test, resulting score gains will be misleadingly large. 

Extant research on the instructional effects of high-stakes testing does not 
examine teachers' responses at the level of detail needed to differentiate among 
these seven categories and is therefore insufficient for purposes of the validation of 
score gains. For example, we know that teachers often reallocate instructional time, 
both within and between subjects, in response to testing (e.g., Koretz, Barron, et al., 
1996; Koretz, Mitchell, et al., 1996; Stecher, 2002). Similarly, research has shown that 
many teachers rely on a variety of test-preparation activities that are consistent with 
the definition of coaching used here (e.g., Koretz, Mitchell, et al.; Koretz, Barron, et 
al.). However, this research offers only a coarse view of the specific elements of 
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instruction that garner increased and decreased attention and the specific test- 
preparation techniques applied. As a result, the findings obtained from this research 
lack the level of detail required to map teacher's responses adequately to the 
inferences that users base on increases in scores. 

Research Questions 

In response to the limitations of extant efforts to evaluate the validity of score 
gains obtained under high-stakes conditions, CRESST has undertaken a multi-year 
effort to develop more productive approaches to this question. The first stage in this 
effort was the development of a conceptual framework for the validation of gains 
(Koretz et al., 2001). This paper represents a second stage of this ongoing effort: a 
pilot study to explore the feasibility of using surveys of teachers to obtain more 
detailed information about their responses to high-stakes testing that could be 
applied to the validation of gains. 

The primary goal of this study is to ascertain whether various types of survey 
questions can begin to capture the types of behavioral responses to testing implied 
by the framework above. Among our principal questions were these: 

• Can teachers understand and respond to survey questions that ask directly 
about the types of behavioral responses implied by the framework above? 

• Can teachers distinguish test items that appeared on their state tests from 
test items that did not appear on their state tests, and if so, can survey 
questions based on test items be used to clarify teachers' behavioral 
responses to testing? 

• Can teachers distinguish content areas emphasized in items on their state 
tests from areas less emphasized, and can survey questions based on these 
content areas be used to clarify teachers' behavioral responses to testing? 

The primary purpose of this study is to provide information that can guide 
future research on the effects of high-stakes testing. It is not primarily intended to 
provide information about the prevalence of particular practices, and the sample is 
not designed to support generalization to populations of interest. The report does 
include information on the distribution of responses to high-stakes testing, but this 
information is intended primarily to provide context (e.g., to establish whether the 
convenience sample used is in some respects similar to those of other studies) and to 
help evaluate the more novel survey items used. The findings should not be 
construed to support strong conclusions about the behaviors of teachers in our 
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study, to support conclusions about the effects of any state testing program, or to 
make inferences about the effects of high-stakes testing more generally. 

Methods 

In this section we describe our research methods: the sample of teachers, the 
data collection instruments, and the analytic approach. 

Study Participants 

This study examined secondary mathematics teachers' responses to the 
Massachusetts Comprehensive Assessment System (MCAS) tests. MCAS is an 
accountability-oriented system of tests first administered in 1998. The MCAS 
assessments are used to produce scores for both students and schools. The study 
design focused on mathematics in Grades 8 and 10 because of the nature of the 
MCAS assessment and accountability system. Although scores at all grades are used 
in school accountability, only the 10 th -grade scores have high stakes for individual 
students. Thus, 10 th -grade teachers might feel more pressure to raise scores and 
therefore might more frequently respond with behaviors that might inflate scores. 
While the sample size would not support a formal comparison of the two grades, 
differences between them could be suggestive. 

The survey focused primarily on the Boston public schools for convenience 
(location) and two reasons of design: It contains a large number of low-scoring 
students, so pressure to raise scores would be high; and it is large enough to provide 
a reasonable sample. Newton, a very high-achieving, affluent, and well-educated 
neighboring suburb, was selected as a second site because it provides a striking 
contrast to Boston, and its very different characteristics might lead to different 
responses by teachers to the MCAS assessment. 

Teachers in Boston were recruited by research assistants at one of four half-day 
mathematics inservice workshops that the district provided in late February and 
early March to support the teachers as they adopted a new curriculum. At these 
workshops, 91 teachers volunteered. Teachers in Newton were recruited by the 
math coordinator of the district. Three teachers initially volunteered. Teachers were 
asked to complete a survey and participate in a telephone interview that would last 
approximately 30 minutes. In exchange, teachers received an honorarium of $100. 






Ten of our initial pool of 94 participants were disqualified because they taught 
only one mathematics class per day. These teachers were excluded because we 
assumed that mathematics was not their central concern, and we expected them to 
be less attentive to preparing students for the mathematics test. Of the remaining 84 
teachers, 51 (61%) completed all data collection described below. Twenty-one (41%) 
of the teachers who completed the data collection taught 8 th -grade mathematics, and 
the remaining 30 (59%) taught 10 th -grade mathematics. The remaining 33 teachers 
who did not complete the study decided not to participate, could not be contacted, 
or completed the survey with sufficient errors that the data could not be used. Only 
3 of the final sample of 51 teachers taught in Newton, so no comparison could be 
drawn between the districts. 

Instruments 

This study focuses primarily on paper-and-pencil questionnaires. In addition, 
we conducted telephone interviews with all teachers after they completed their 
surveys. Only a few results from the interview data are described here. 

The questionnaire included four sections. The first section gathered 
background information, including the number and titles of courses taught. 
Teachers were asked to consider the first 8 th ' (or 10 th ') grade mathematics class of the 
day the "target" class and were instructed to respond to the remaining questionnaire 
items based on this class. The first section also asked teachers to rate the math 
achievement levels of their schools in relation to the state and of students in the 
target class in relation to the school. In the second section, we asked several types of 
questions about teachers' responses to the MCAS test. These questions had several 
purposes. A few were designed to put the teachers' responses into context - for 
example, questions about the degree of pressure respondents felt to raise scores and 
the amount of flexibility they had to respond. A few were similar to those used in 
earlier surveys and could help to clarify whether the small convenience sample used 
in this pilot led to anomalous responses. 

This section included one multi-part question about the resources teachers 
used for test preparation (e.g., old test items and the Massachusetts Department of 
Education's test preparation website). It also included a long, multi-part question 
that asked how respondents' teaching is different from what it would be without 
MCAS. The prompts were intended to categorize common test-preparation activities 
consistently with the framework described above. For example, one part asked how 
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much respondents shifted emphasis among strands or topics in the mathematics 
curriculum; another asked how much they tailored instruction to "particular styles 
and formats of problems in MCAS." 

One of the questions motivating this study was what types of survey questions 
would be useful for eliciting information about the types of test preparation relevant 
to validation. It was not clear that questions of the sort just described-that is, 
questions relying on verbal descriptions of categories of response - would be clear 
enough to respondents to elicit useful information. We therefore included a section 
of novel questions based on actual test items from MCAS and NAEP. The rationale 
for including items from NAEP is discussed below. Six MCAS and four NAEP items 
were included. Teachers were given no information about the source of the items, 
and the textual parts of the items were reformatted so that format would not 
provide a hint about the source. Teachers were asked an identical set of questions 
about each item. These included whether they recognized each item and, if so, from 
where; what skills or knowledge they thought the item required; and whether they 
taught the content of the item. Additional follow-up questions were asked, 
depending upon teachers' responses to the question about whether they taught the 
item. The final section of the questionnaire presented seven specific content areas 
(e.g., "use proportional thinking to solve problems") and asked teachers a series of 
questions about each, including whether or not they taught that content area. A copy 
of one of the survey forms is included as the Appendix. 

As discussed earlier, a common consequence of high-stakes testing is a 
tendency for teachers to shift time and effort toward tested content and away from 
material that is not included in the test. One of the objectives of this effort was to 
examine methods for exploring this phenomenon. With only one year of data 
collection, we were not able to observe changes in emphasis directly. To explore 
changes in emphasis, we followed two strategies. First, we asked a variety of 
questions about how MCAS influenced teachers' practice (e.g., how their practice 
would change if MCAS were not given). Second, we asked a series of questions 
based on individual test items or descriptions of content categories that had varying 
degrees of emphasis in MCAS. These were designed to focus teachers' responses 
about their practice on specific content and styles of items. 

To create the latter group of questions, it was necessary to identify item types 
and content areas that had varying emphasis on the MCAS. As one step in this 
process, we compared the NAEP Mathematics Framework to both released NAEP 
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items and three years of released MCAS test forms. We categorized each item on 
each test form in terms of the elements of the framework. The authors categorized 
items independently and then reconciled differences. This process yielded a tally 
indicating how frequently each element of the NAEP framework was the focus of 
NAEP and MCAS items. As a guide to selecting items and content areas for 
inclusion in the report, we noted the framework elements that appeared frequently 
in one of the tests and rarely or never in the other. 

As a second step, both authors examined both item sets independently to find 
patterns in the styles of items presented. Starting with the eighth-grade MCAS, we 
identified 13 types of items that were repeated across years (e.g., problems involving 
numerical and geometric series, problems that required mentally cutting open a 
three-dimensional shape, and problems entailing very simple rotations of figures), 
as well as two items that were not repeated but seemed to both authors to be quite 
unusual. We examined NAEP released items both to see if item types we had 
identified in examining MCAS appeared in NAEP and to find types of items that 
appeared in NAEP but not in MCAS. For example, we noted that NAEP included 
items with irregular polygons, while the MCAS forms we used presented only 
regular polygons. Conversely, two of three MCAS forms presented algebra items in 
a pictorial format not used in NAEP. 

Based on these two steps, we selected 6 items from MCAS and 4 from NAEP to 
include in the survey in each grade. In addition, for the S^-grade survey we selected 
5 content areas to represent MCAS and 2 to represent NAEP. For the 10 th -grade 
survey, we selected 4 MCAS and 3 NAEP content areas. 

Although the resulting sets of items and content areas are in some sense 
characteristic of the two assessments, by design they are not representative of them. 
Our goal was to determine whether surveys can be used to identify ways in which 
teachers focus their test-preparation efforts on details that recur in the tests for 
which they are held accountable. Therefore, we attempted to identify content areas 
and types of items that received particular emphasis in the high-stakes test, as well 
as commonly taught aspects of content and problem types that are given little or no 
emphasis on that test, in order to determine whether we could discern shifts in 
teachers' emphasis corresponding to these. For this purpose, types of items common 
to both assessments are not useful, even though such items must be included to fully 
represent the test and therefore must be included in other types of work evaluating 
the validity of gains. 
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Findings 

In this section we first present basic summary information on the background 
and general test preparation questions from the survey, followed by a discussion of 
responses to the specific item and content questions. Because the type and 
magnitude of test-related pressure is likely to differ for 8 th - and lO^-grade teachers, 
we present most results separately by grade. 

Background Questions 

The results presented in this section provide information on the teachers in the 
sample and the contexts in which they were teaching. Among the 30 10 th -grade 
teachers, all of whom taught in the Boston Public Schools, the average amount of 
teaching experience was 12 years, with experience ranging from 1 to 31 years. Three 
participants taught special education classes primarily. 

The 8 th -grade sample included 21 teachers, 3 from Newton and the remaining 
18 from Boston. As with the 10 th -grade sample, the average experience level was 12 
years, and experience ranged from 1 to 35 years. Only 1 teacher in this group 
reported teaching special education. 

As shown in Table 1, a majority of teachers at each grade reported that their 
school was performing at or below the state average. This is somewhat consistent 
with the generally low performance of Boston schools, though based on the state's 
published test scores, nearly all teachers should have reported below-average school 
performance. All three Newton teachers reported achievement levels that were 

Table 1 



Percentages of Teachers Reporting Above-Average, Average, or Below- 
Average Achievement of the School as Compared With Other Schools 
in the State ( N in Parentheses) 





Grade 8 


Grade 10 


Substantially below average 


29(6) 


40 (12) 


Moderately below average 


29(6) 


30 (9) 


About average 


19 (4) 


17(5) 


Moderately above average 


5(1) 


7(2) 


Substantially above average 


19 (4) 


7(2) 
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substantially above average, which is consistent with published results for that 
district. 

The fact that teachers' perceptions of their school's achievement were not 
always consistent with published test scores suggests that teachers are either 
unaware of how their school ranks or that they are using criteria other than test 
scores to evaluate achievement. Either way, these results indicate the importance of 
verifying teacher-reported information about achievement if the purpose of the data 
collection is to understand how well students are actually performing. 

Teachers were also asked to compare the math achievement level of the class 
about which they were responding with that of other classes in the school. A 
majority of 8 th -grade teachers reported average achievement levels, whereas only a 
third of 10 th -grade teachers did so (see Table 2). This difference was expected, given 
the higher proportion of tracked classes in the 10 th grade. Unfortunately, we have no 
information with which to verify the accuracy of these reports. If this information 
were important for subsequent analysis, it would be worth gathering additional data 
to corroborate teachers' responses to this question. 

Questions About General Responses to MCAS 

The first of the questions addressing general responses to MCAS asked 
teachers how much pressure they felt to improve their students' performance on the 
MCAS Mathematics test. Because the nature and amount of pressure is likely to vary 
across districts, we omitted the three 8 ,h -grade Newton teachers from this table. 
Slightly more than half of the teachers at each grade reported feeling "a great deal" 

Table 2 



Percentages of Teachers Reporting Above-Average, Average, or 
Below-Average Achievement of the Class as Compared With Other 
Classes in the School (N in Parentheses) 





Grade 8 


Grade 10 


Substantially below average 


5(1) 


10 (3) 


Moderately below average 


16(3) 


27(8) 


About average 


68 (13) 


33 (10) 


Moderately above average 


11(2) 


20(6) 


Substantially above average 


0 


10 (3) 



of pressure to improve their students' scores (Table 3). Only two teachers in 10 th 
grade reported feeling no pressure, and none of the Boston S^-grade teachers gave 
this response. These results suggest that even though the formal stakes were higher 
at 10 th grade at the time of the survey, teachers at 8 th grade were as likely to 
experience significant pressure for score improvement. While these findings suggest 
that our sample of teachers felt considerable pressure as a result of MCAS, 
comparison with other studies suggests that these findings represent a relatively 
modest degree of pressure, and that in turn may help explain some of the relatively 
modest patterns shown in our subsequent survey results. For example, Koretz, 
Barron, et al. (1996) asked Kentucky teachers whether they agreed with the 
statement that teachers were under "undue pressure to improve students' 
performance on KIRIS" (the state assessment used for accountability purposes). 
Almost all (98%) of a state-representative sample of teachers agreed with this 
statement, and 80% strongly agreed with it. 

Because of the differing level of stakes for students at the two grades, we 
expected 10 th -grade teachers to report higher levels of pressure than S^-grade 
teachers. It is possible that the question as written was not sufficiently sensitive to 
differences that might exist. On the other hand, perhaps 8 th -grade teachers do 
experience pressure despite the lack of high stakes. Case studies of Florida schools 
suggest that simply publishing scores and assigning grades or ratings to schools 
creates pressure for school staff to maintain or increase their standing (Goldhaber & 
Hannaway, 2001). To more fully understand these results and the absence of any 
clear difference between the grades it would be necessary to gather information on 
the sources of the pressure. 

Table 3 



Percentages of Teachers Reporting Feeling Pressure to 
Improve Their Students' Scores on the MCAS Mathematics 
Test (N in Parentheses) 





Grade 8 


Grade 10 


None 


0 


7(2) 


A small amount 


17(3) 


17(5) 


A moderate amount 


26(5) 


23(7) 


A great deal 


53 (10) 


53 (16) 
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15 



The second question in this section asked teachers to describe how well MCAS 
was aligned with the curriculum in place in their schools. Table 4 shows that nearly 
60% of the 10 th -grade teachers and slightly more than half of the 8 th -grade teachers 
reported that MCAS was "fairly well" aligned with their school's curriculum. 
Approximately one third of teachers at each grade said MCAS was poorly aligned 
with their school's curriculum, either because it included material not in the 
curriculum or because it omitted material that is included in the curriculum. 

The next set of questions was designed to gather information on test- 
preparation activities. The first question in this set asked teachers to describe how 
much emphasis they placed on various test-preparation resources (Tables 5 and 6). 
The specific resources listed in this question were chosen based on pilot interviews 
with teachers and inspection of the Massachusetts Department of Education's 
website. Among the resources listed, teachers in both grades were most likely to 
report strong emphasis on old MCAS items and other sample MCAS items (e.g., 
those available in materials published by the Department of Education). Other 
resources, such as the Princeton Review's Cracking the MCAS, the Boston Public 
Schools' "Tool Kit," and the Massachusetts Department of Education's MCAS 
website, were used infrequently, especially by 8 th -grade teachers. The Princeton 
Review book was designed for the 10 th -grade test, so it is not surprising that hardly 
any 8 th -grade teachers said they used this. In general 10 th -grade teachers reported 
more emphasis and a greater variety of test-preparation materials than 8 th -grade 
teachers, but teachers in both grades relied most heavily on actual test items. This 
indicates that the test itself has a strong influence on curriculum and instruction, a 
finding that is consistent with other work (Stecher, 2002). It suggests substantial 
reallocation and coaching, but without providing the detail needed to apply the 
framework suggested by Koretz et al. (2001). 

Table 4 



Percentages of Teachers Reporting Alignment Between School's Curriculum and 
MCAS (N in Parentheses) 





Grade 8 


Grade 10 


Poorly, because MCAS includes considerable material 
that is not in our curriculum 


19(4) 


24(7) 


Poorly, because MCAS omits considerable material 
that is in our curriculum 


10(2) 


10(3) 


Fairly well, although the overlap is not complete 


52 (11) 


59 (17) 


Very well 


19(4) 


7(2) 
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Table 5 



Percentages of Teachers Reporting Emphasis on Test-Preparation Resources, Grade 8 
(N in Parentheses) 





No emphasis 


Slight 

emphasis 


Moderate 

emphasis 


Great 

emphasis 


Old MCAS items 


0 


19(4) 


24(5) 


57 (12) 


Other sample MCAS 
test items 


5(1) 


20 (4) 


30 (6) 


45(9) 


MCAS scoring rubrics 


10(2) 


29(6) 


33 (7) 


29 (6) 


The Princeton Review's 
Cracking the MCAS 


95 (18) 


5(1) 


0 


0 


The MCAS2003.com 
website 


72 (13) 


17(3) 


11(2) 


0 


The BPS "Tool Kit "* 


73(11) 


13 (2) 


7(1) 


7(1) 


Other commercial test- 
prep books or materials 


42(8) 


32 (6) 


26 (5) 


0 


Other DOE or BPS test- 
preparation materials* 


47(9) 


16(3) 


11(2) 


26(5) 



*For Newton teachers, the BPS Took Kit item was not presented, and the final item 
mentioned only DOE materials. Percentages for this option are based only on the 18 
Boston teachers. 



Table 6 

Percentages of Teachers Reporting Emphasis on Test-Preparation Resources, Grade 10 
(N in Parentheses) 





No emphasis 


Slight 

emphasis 


Moderate 

emphasis 


Great 

emphasis 


Old MCAS items 


3(1) 


10(3) 


33 (10) 


53 (16) 


Other sample MCAS 
test items 


7(2) 


10(3) 


43 (13) 


40 (12) 


MCAS scoring rubrics 


14(4) 


39(11) 


25 (7) 


21 (6) 


The Princeton Review's 
Cracking the MCAS 


66 (19) 


24(7) 


3(1) 


7(2) 


The MCAS2003.com 
website 


43 (13) 


47 (14) 


10(3) 


0 


The BPS "Tool Kit" 


47 (14) 


30 (9) 


20(6) 


3(1) 


Other commercial test- 
prep books or materials 


38(11) 


38 (11) 


10(3) 


14 (4) 


Other DOE or BPS test- 
preparation materials 


37 (11) 


23(7) 


30 (9) 


10(3) 
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To understand the degree to which teachers had freedom to engage in test 
preparation activities, we asked them how much flexibility they had to modify their 
curriculum and instruction in response to MCAS. Table 7 presents these results. We 
excluded the three Newton teachers from this table because of fundamental 
differences in the district contexts in which they are operating. Two thirds of 
teachers at each grade reported having very little flexibility, and no 10 th -grade 
teachers said they had a great deal. Among the three Newton teachers, one reported 
having a great deal of flexibility, and the other two reported a moderate amount. 
Teachers who chose "very little, but for some other reason" were asked to specify 
the reason. A total of six teachers chose this option, and most of those who provided 
an explanation indicated that the reason was related to curricular requirements (e.g., 
"my school required all geometry teachers to spend the first three months teaching 
more algebra 1"). It is not clear why these teachers chose the second rather than the 
first option, but the responses suggest these two options could be combined without 
loss of information. 

The lack of flexibility reported by the Boston teachers - which many of them 
reiterated in interviews - is important for evaluating our methods, which were 
designed with the expectation that teachers had considerable flexibility. We return 
to this in the final Discussion section. 

In the next question, we attempted to capture information about the types of 
test preparation described in Koretz et al. (2001) and discussed in the introduction to 
this report. Because not all teachers had taught prior to the introduction of MCAS in 
1998, we were unable to ask about specific changes they had made in response to the 
introduction of a high-stakes test. Instead, we asked teachers about ways in which 
their teaching might be different because of MCAS than it would be without MCAS. 
We developed several categories of responses that corresponded to some of the 

Table 7 



Percentages of Teachers Describing Flexibility to Modify Their Curriculum 
and Instruction in Response to MCAS (N in Parentheses) 





Grade 8 


Grade 10 


Very little, because my school or district 
requires me to adopt a specific curriculum 


67 (12) 


67 (20) 


Very little, but for some other reason 


6(1) 


10(3) 


A moderate amount 


11(2) 


23(7) 


A great deal 


17(3) 


0 
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types of test preparation discussed at the beginning of this paper and in Koretz et al. 
(2001). As we discuss later in the section on questions about specific test items, we 
found the task of describing the different forms of test preparation to be challenging. 
In particular, distinguishing among some of the subtle types of reallocation and 
coaching was difficult, and we found from our interviews that teachers did not 
always understand these distinctions from the questions we posed. Thus, although 
the responses regarding the broad categories of test preparation reported here are 
likely to be reasonably accurate, responses to some of the specific parts may be less 
so. 

The responses of eighth-grade teachers suggest that MCAS substantially 
influenced instruction, in both desired and undesired ways (see Table 8). Almost all 
S^-grade teachers reported placing a greater emphasis on writing in mathematics. At 
the other extreme, about two thirds of the teachers reported that they had not 
assigned more homework or more difficult homework. A majority of teachers 
reported some reallocation and coaching. Their responses showed some 
differentiation among types of reallocation and coaching, which is a positive sign 
about the potential utility of questions of this sort, but the small number of 
respondents make these results only suggestive. Ranking the items in terms of the 
percentages of teachers who responded that they used each method either "not at 
all" or "a great deal" yielded fairly consistent patters. Apart from assigning 
homework, the types of preparation most often cited as not used at all were shifting 
time from material not in the local curriculum and finding more effective teaching 
methods. Both would generally be considered desirable responses. Almost half said 
that they had relied not at all on finding more effective methods of teaching. Apart 
from placing more emphasis on writing, the approach used a great deal was giving 
more emphasis to "test-taking strategies, such as 'plugging in' and 'process of 
elimination/" which we consider undesirable because it will not provide the 
students with greater understanding of mathematics. Most teachers reported that 
they "look for patterns in the ways that MCAS presents a given type of content and 
emphasize those in [their] instruction" — an aspect of coaching that we were 
particularly interested in exploring — but a majority of teachers said they relied only 
"somewhat" on this approach. 
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Table 8 



Percentages of Teachers Reporting That Ways in Which Their Teaching Is Different Because of 
MCAS, Grade 8 (N in Parentheses) 





Type of test 
preparation 


Not at 
aH 


Somewhat 


A great 
deal 


I spend more time teaching mathematics content 
(e.g., by replacing non-instructional activities with 
instruction on mathematics). 


Teaching 

more 


37 (7) 


42(8) 


21 (4) 


I assign more homework or more difficult 
homework. 


Teaching 

more 


60 (12) 


35 (7) 


5(1) 


MCAS has prompted me to find more effective 
teaching methods (more effective in general, not 
specifically targeted to MCAS). 


Working 

more 

effectively 


43(9) 


43(9) 


14 (3) 


I focus more on the Massachusetts Curriculum 
Framework Content Standards in mathematics. 


Reallocation 


14 (3) 


57 (12) 


29(6) 


I shift emphasis among topics or strands of the 
mathematics curriculum to focus more on material 
emphasized in MCAS (e.g., shifting instructional 
time from coordinate geometry to plan geometry 
or vice-versa). 


Reallocation 


29(6) 


57 (12) 


14 (3) 


I focus more on the aspects of performance that are 
stressed by the MCAS scoring rubrics. 


Reallocation 


16(3) 


58 (11) 


27(5) 


I place a greater emphasis on writing with 
mathematics. 


Reallocation 


10(2) 


52(11) 


38(8) 


I shift emphasis from other mathematics material 
not in the local curriculum to focus more on material 
emphasized in MCAS. 


Alignment, 

reallocation 


48 (10) 


43(9) 


10(2) 


I look for patterns in the ways that MCAS presents 
a given type of content and emphasize those in my 
instruction (e.g., using coordinate geometry to 
present algebra problems). 


Coaching 


24(5) 


62 (13) 


14(3) 


I look for particular styles and formats of problems 
in MCAS and emphasize those in my instruction 
(e.g., using unusual pictures or symbols to present 
algebra problems; using specific key phrases). 


Coaching 


10(2) 


62 (13) 


29(6) 


I place more stress than I would on mathematics 
test-taking strategies, such as "plugging in" and 
"process of elimination." 


Coaching 


5(1) 


57 (12) 


38 (8) 


I spend more time teaching and encouraging 
generic test-taking strategies such as time 
management or eating a good breakfast. 


Coaching 


33 (7) 


57 (12) 


10(2) 
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The responses of lO^-grade teachers were in many respects similar but differed 
in a few suggestive ways (see Table 9). Increased emphasis on writing was again the 
most commonly reported approach. Eighth-grade teachers placed somewhat more 
emphasis on test-taking strategies than lO^-grade teachers, whereas lO^-grade 



Table 9 

Percentages of Teachers Reporting That Ways in Which Their Teaching Is Different Because of MCAS, 
Grade 10 (N in Parentheses) 





Type of test 
preparation 


Not at all 


Somewhat 


A great 
deal 


I spend more time teaching mathematics content 
(e.g., by replacing non-instructional activities with 
instruction on mathematics). 


Teaching 

more 


38 (11) 


31 (9) 


31 (9) 


I assign more homework or more difficult 
homework. 


Teaching 

more 


53 (16) 


27 (8) 


20 (6) 


MCAS has prompted me to find more effective 
teaching methods (more effective in general, not 
specifically targeted to MCAS). 


Working 

more 

effectively 


37 (11) 


37 (11) 


27 (8) 


I focus more on the Massachusetts Curriculum 
Framework Content Standards in mathematics. 


Reallocation 


27 (8) 


37 (11) 


37 (11) 


I shift emphasis among topics or strands of the 
mathematics curriculum to focus more on material 
emphasized in MCAS (e.g., shifting instructional 
time from coordinate geometry to plan geometry or 
vice-versa). 


Reallocation 


13(4) 


47 (14) 


40 (12) 


I focus more on the aspects of performance that are 
stressed by the MCAS scoring rubrics. 


Reallocation 


28(8) 


52 (15) 


21 (6) 


I place a greater emphasis on writing with 
mathematics. 


Reallocation 


7(2) 


57 (16) 


36 (10) 


I shift emphasis from other mathematics material not 
in the local curriculum to focus more on material 
emphasized in MCAS. 


Alignment, 

reallocation 


37 (11) 


40 (12) 


23(7) 


I look for patterns in the ways that MCAS presents a 
given type of content and emphasize those in my 
instruction (e.g., using coordinate geometry to 
present algebra problems). 


Coaching 


20(6) 


53 (16) 


27 (8) 


I look for particular styles and formats of problems 
in MCAS and emphasize those in my instruction 
(e.g., using unusual pictures or symbols to present 
algebra problems; using specific key phrases). 


Coaching 


27(8) 


30(9) 


43 (13) 


I place more stress than I would on mathematics test- 
taking strategies, such as "plugging in" and "process 
of elimination." 


Coaching 


23(7) 


40 (12) 


37 (11) 


I spend more time teaching and encouraging generic 
test-taking strategies such as time management or 
eating a good breakfast. 


Coaching 


40 (12) 


43 (13) 


17(5) 
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teachers reported more reallocation among mathematics topics. Tenth-grade 
teachers showed less differentiation among types of test preparation, but this could 
easily be a result of the small sample sizes. In general, there was a slight tendency 
for lO^-grade teachers to report more changes in their instruction in response to 
MCAS than 8 lh -grade teachers, but the difference was not large. This is not 
surprising given the similar levels of pressure reported by the two groups of 
teachers. 

The next question asked teachers to describe the magnitude of MCAS gains at 
their schools between 2000 and 2001. Those who responded that their schools' scores 
had increased were then asked to rate the importance of a set of factors that may 
have contributed to those gains. This provides important contextual information for 
interpreting teachers' responses regarding test preparation activities. Table 10 shows 
that while most teachers reported some gain, only 10 th -grade teachers said their 
schools' scores had risen "a great deal." This difference is consistent with published 
test scores, which show that 8 th -grade scores did not rise much but 10 th -grade scores 
did. However, there appears to be a tendency for teachers to overestimate the 
magnitude of gain, based on the fact that more than half of the 8 th -grade teachers 
reported some gain. 

Table 11 summarizes the responses to the question about attributions of gains. 
Teachers who responded "not at all" or "I don't know" were instructed to skip this 
question, so the number of respondents represented in these tables is small. We, 
therefore, combined results across the two grades, although the responses did differ 
somewhat between the grades. 1 Responses are sorted by the percentage of teachers 
who attributed a great deal of impact to each factor. 

Table 10 



Percentages of Teachers Reporting Increase in MCAS Scores 
Between Spring 2000 and Spring 2001 (N in Parentheses) 





Grade 8 


Grade 10 


Not at all 


10(2) 


10 (3) 


A small amount 


24(5) 


20 (6) 


A moderate amount 


38(8) 


17(5) 


A great deal 


0 


37(11) 


Don't know 


29(6) 


17(5) 



1 10 th -grade teachers were more likely than 8 th -grade teachers to attribute gains to test-preparation 
activities, to increased familiarity with the content of MCAS, and to motivation. 



The attribution of gains reported by many teachers suggests the possibility of 
inflated score gains. Broad improvements in students' knowledge and skills - that is, 
improvements that are unarguably a source of valid gains - were the least often 
cited as having a great deal of impact. Improvements in the knowledge and skills 
emphasized in MCAS, which could result in either meaningful or inflated gains, 
were roughly in the middle of the list. The factors cited most often as having a great 
deal of impact were increased familiarity with the MCAS assessment and work with 
practice tests and other test preparation materials, which could lead to test-specific 
gains that would not generalize well. Despite differences in the assessment 
programs, the grades surveyed, and the unrepresentativeness and very small size of 
the convenience sample in this study, these percentages are remarkably similar to 
responses obtained in large, state-representative samples of teachers in earlier 
studies of Kentucky and Maryland (Koretz, Barron, et al., 1996; Koretz, Mitchell, et 
al., 1996). This may suggest commonalities in teachers' responses to high-stakes 
testing, and it provides a measure of assurance that responses to this pilot study are 
not badly distorted by the nature of the sample. 

Table 11 



Percentages of Teachers Reporting That Various Factors Had a Moderate Amount 
or Great Deal of Impact on Scores, Both Grades (N in Parentheses) 





A moderate 
amount 


A great deal 


Increased familiarity with the MCAS assessments 


33 (12) 


56 (20) 


Work with practice tests and other preparation 
materials 


36 (13) 


53 (19) 


Increased student motivation 


31 (11) 


39 (14) 


Improvements in students' mastery of 
knowledge and skills that are emphasized in 
MCAS 


36 (13) 


36 (13) 


Improved test-taking skills 


42(15) 


25(9) 


Differences between student cohorts from year to 
year 


19 (7) 


14 (5) 


Broad improvements in students' knowledge and 
skills 


44(16) 


11 (4) 



Note. Respondents include only those teachers who said their school's scores rose 
between 2000 and 2001. 
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Item-Based Questions 



The attributes of test items upon which teachers can focus to raise scores — what 
Koretz, et al. (2001) called substantive and non-substantive performance 
elements — are diverse and difficult to describe clearly. We found it difficult to 
translate these concepts into common language in writing the survey reported here, 
and some of our respondents found it difficult to understand the terminology used. 
Tables 8 and 9 presented the results for question 6, in which teachers were asked to 
characterize ways in which their teaching is different than it otherwise would be 
because of MCAS. Among the choices were "I look for patterns in the ways that 
MCAS presents a given type of content and emphasize those in my instruction" and 
"I look for particular styles and formats of problems in MCAS and emphasize those 
in my instruction." These phrases all represented specific attributes of MCAS items 
identified by the authors, and although we provided examples to clarify the phrases, 
our interviews suggest that some respondents could not understand the differences 
between them. 

We created questions based on specific test items in an effort to circumvent this 
difficulty. Our goal was to test questions that illustrate these attributes concretely 
with test items rather than identifying them with general labels such as "styles of 
problems." Six of the 10 items administered to survey respondents at each grade 
level were items from past MCAS assessments that had been chosen by the authors 
to represent attributes that the authors found characteristic of MCAS. These 
attributes included item format, styles of presenting material, and specific content. 
The remaining four items were publicly released NAEP items that did not share 
these characteristics. Because NAEP is not administered in the 10 th grade, we 
selected for the lO^-grade sample NAEP items administered in both Grade 8 and 
Grade 12. Each item was followed by a series of questions asking the teacher if he or 
she recognized the item and if so from what source, what students need to know or 
be able to do to answer the question, whether the teacher taught this content, and 
why or why not (see Appendix). Although the specific questions were different, this 
general approach has some similarities to the item-based methods used to explore 
opportunity to learn in the First and Second International Mathematics Studies (see 
Floden, 2002). 

Examples of sample items. An example of an MCAS item selected on the basis 
of content is shown in Figure 1. This item was chosen because it is flagged in the 
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Princeton Review's Cracking the MCAS Grade 10 Math (Princeton Review, 2000), a 
test-preparation book that could be located via a state-sponsored website but that 
our survey showed to be emphasized relatively little by respondent teachers (see 
Tables 5 and 6). Specifically, in a section labeled "Special Triangle Rules," Cracking 
the MCAS stated that "one triangle rule that is often tested on the MCAS exam is the 
third side rule. The rule is: The sum of every two sides of a triangle must be greater 
than the third side" (Princeton Review, p. 52). 

Item H . If the perimeter of an isosceles triangle is 24 cm, which of the following cannot be 
the base? 

A. 4 cm 

B. 6 cm 

C. 10 cm 

D. 12 cm 

Figure 1. A sample 10 ,h -grade MCAS item. 

Most of the items chosen for inclusion in the survey, however, had more 
obvious attributes that made them (in the opinion of the authors) characteristic of 
the MCAS. For example, we found that the MCAS tests included algebra problems 
with unconventional representations of unknowns. One example, in which 
unknowns are represented as polyhedra and equality is represented by equal levels 
on a balance beam, is shown in Figure 2. Two such items were found in the three 
eighth-grade MCAS tests we examined. 



Item F. Use the balance scales below to answer the question below. 




Figure 2. A sample 8 lh - grade MCAS item. 
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The MCAS tests we examined also included a number of items that we 
considered to be tapping spatial visualization skills, a type of item lacking in many 
other assessments. An example from the lO^-grade survey is shown in Figure 3. The 
test also included a number of items focusing on numerical or geometric series. In 
fact, this was one of the most common problem types we identified, and one that 
many teachers identified as representative of MCAS in our interviews. When asked 
two open-ended questions — "Have you noticed any aspects of mathematics that get 
particularly strong emphasis on MCAS? If so, what are they?"; and "Have you 
noticed any specific content that appears frequently in MCAS items?" — one-fourth 
of our teachers mentioned patterns or sequences in response to one or both 
questions. An example of this type of item is shown in Figure 4. 

Item C . Use the figure below to answer the next question. 




If the above figure is folded into a cube, which of the following solids will be formed? 




Figure 3. A sample 10 th -grade MCAS item. 
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Item D . Each arrangement in this pattern is made up of tiles. 




How many tiles will be in the 6 th arrangement in the pattern? 

Figure 4. A sample S^-grade MCAS item. 

Several of the MCAS forms we reviewed included a long open-response 
question about simultaneous equations, typically presented as a comparison 
between two billing or cost structures. We included one example, shown in Figure 5. 

Item F . Use the chart below to answer the question below. 



Billing Plans for Cellular Phone 




Basic charge 
per month 


Number of 
free minutes 
per month 


Charge per 
minute 


Plan 1 


$16 


0 


$0.35 


Plan 2 


$30 


50 


$0.15 

after free minutes 
are used 



Mr. Chrostowski is choosing one of the billing plans shown above for his cellular phone. He estimates 
that he will use the phone less than 50 minutes per month. 

A. If he chooses Plan 1 and uses the phone exactly 50 minutes in one month, what will his bill be for 
that month? 

B. Suppose that he chooses Plan 1 and uses the phone m minutes in one month. Write an equation for 
his total bill, B, for that month. 

C. On the grid in your Student Answer Booklet, construct a graph that shows the monthly bills for 
Plan 1 for between 0 and 50 minutes of calls. 

D. Using your equation or graph, find the number of minutes of phone use for which the two plans 
cost the same. Show or explain how you found your answer. 

Figure 5. A sample lO^-grade MCAS item. 



One additional class of MCAS items we could only characterize as logic 
problems or "puzzles." Many of these entailed mathematical content sufficiently 
complex that most students would end up solving them (or attempting to solve 
them) by trial and error rather than by applying a clear body of mathematical 
knowledge or skills. An example is given in Figure 6. This item is technically about 
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network theory, but our expectation, confirmed by our survey, was that few teachers 
in our sample would present it that way. When asked in an open-response question 
"What must a student know or be able to do in order to answer this item correctly?" 
30 teachers provided answers for this item, and of those, only 3 mentioned networks 
or network theory in their answers. The remaining answers were highly diverse and 
mostly focused on generic skills or trial and error (e.g., "process of elimination," 
"trial and error," "problem-solving techniques," "connect points in a variety of ways 
until it matches the criterion," "logic.") 



Item A . Use the map below to answer this question. 




Mr. Hendricks operates a snowplow for the Department of Public Works (D.P.W.). He found 
that he can 

• begin snowplowing at the D.P.W. 

• plow every street shown on the map above without going over any street more than 
once, and 

• end at his home. 

Where is his home located? 

A. at A 

B. at B 

C. at C 

D. at D 

Figure 6. A sample lO^-grade MCAS item. 

The NAEP items were chosen because they lacked attributes we had identified 
as characteristic of MCAS and had other attributes that seemed rare or entirely 
lacking in MCAS. For example, we noted that all of the MCAS problems involving 
polygons that we located entailed regular polygons and calculation rather than 
estimation of area or perimeter. 2 Therefore it seemed logical to include the NAEP 
item shown in Figure 7, which shows an irregular polygon and entails estimation. 

2 In the interviews conducted after these surveys were completed, teachers were asked "Have you 
noticed any important aspects of mathematics that get little or no emphasis in MCAS? If so, what are 
they?" A single 10 th -grade teacher listed irregular polygons. 





Note that the content of this item is aligned with the Massachusetts standards. That 
is, there is nothing in the Massachusetts Curriculum Frameworks indicating that 
skills should extend only to regular polygons. For example, the Mathematics 
Curriculum Framework lists under Exploratory Concepts and Skills for Grades 7-8: 
"Investigate formulas to determine the circumference and area of circles, and the 
perimeter and area of polygons" (Massachusetts Department of Education, 2000, 
p. 67). 

Item E . Use the unit of length below to estimate the perimeter of the figure shown. Between which 
two consecutive whole-number units does the perimeter he? 




Answer: Between and 

Figure 7. A sample 8 th -grade NAEP item. 

Because trigonometry has been removed from the MCAS standards for Grade 
10 but is a standard part of the mathematics curriculum in many schools, we 
decided to include a NAEP item that assessed simple trigonometry (see Figure 8). 
Note that reallocation away from the content of this item would not be a threat to 
the inferences that MCAS is designed to support, although it might be a threat to the 
inferences users actually draw, since many users are probably unfamiliar with the 
framework and assume that trigonometry is part of a typical lO^-grade mathematics 
curriculum. However, it is an interesting opportunity to test the extent of 
reallocation. 
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Item E . 




In right triangle ABC above, cos A = 

A) 3/5 

B) 3/4 

C) 4/5 

D) 4/3 

E) 5/3 

Figure 8. A sample 10 th -grade NAEP item. 



We also included NAEP items that addressed content specifically noted in the 
Massachusetts Curriculum Frameworks but that lacked any particular attribute 
characteristic of MCAS. For example, the Massachusetts Mathematics Framework 
states that "In high school, students... learn to choose representative samples and 
identify biases in the samples and survey questions" (Massachusetts Department of 
Education, 2000, p. 16). We did not find many MCAS items directly addressing this, 
but the NAEP item in Figure 9 clearly does. 



Item C . A poll is being taken at Baker Junior High School to determine whether to change the 
school mascot. Which of the following would be the best place to find a sample of 
students to interview that would be most representative of the entire student body? 

A) An algebra class 

B) The cafeteria 

C) The guidance office 

D) A French class 

E) The faculty room 

Figure 9. A sample 8 th -grade NAEP item. 



Teacher-level results. The responses to these item-based questions can be 
looked at in two ways: by examining the distribution of responses across teachers. 
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and by looking for patterns across items. In this section, we look at the former. The 
latter is addressed in the subsequent section. 

Teachers' responses to these questions are shown in "dit plots" such as Figure 
10. Dit plots, while unfamiliar to many readers, are a good way to show the entire 
distribution of cases when samples are small. In this figure, the scale on the X-axis 
represents the proportion of items a teacher reported recognizing from MCAS. Each 
teacher is represented in this figure by two symbols, a + and a o. The + indicates the 
proportion of NAEP items the teacher incorrectly identified as being from MCAS, 
and the o indicates the proportion of MCAS items the teacher correctly identified as 
being from MCAS. For example, if a teacher did not identify a single NAEP item 
incorrectly as coming from MCAS, that teachers + would be at a value of zero. If 
teachers were completely accurate, all of the + symbols would stack up at 0, and all 
the o signs would stack up at 1.0. Note that as an artifact of the way we posed these 
questions, teachers will seem more accurate in responding to NAEP items than in 
responding to MCAS items. This is because teachers who fail to recognize test items 
in general will tend to answer "no" when asked whether they recognize any item. 
Thus they will correctly not identify NAEP items as coming from MCAS but will 
incorrectly fail to identify MCAS items. 

Teachers were able to distinguish with moderate accuracy between items that 
were and were not take from previous forms of MCAS, which suggests that they pay 
substantial attention to the details of the test. Most of the 8 th -grade teachers did not 
identify any of the NAEP items as coming from MCAS (see Figure 10). They less 
commonly recognized actual MCAS items. For example, four teachers recognized 
roughly two thirds of the MCAS items, and six recognized fewer. Grade 10 teachers 
appear to differentiate slightly less successfully between MCAS and NAEP items 
(see Figure 11), but the small numbers of teachers and items makes this uncertain. 
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Figure 10. Proportion of items "recognized" as being from 
MCAS by each teacher, Grade 8 (+ symbol indicates NAEP 
items; o symbol indicates MCAS items). 




Proportion of items recognized from M( 



Figure 11. Proportion of items "recognized" as being from MCAS 
by each teacher. Grade 10 (+ symbol indicates NAEP items; o 
symbol indicates MCAS items). 



O 

ERIC 



30 



32 



If teachers attempt to reallocate their instruction to align it better with the 
content of test items, the alignment of their instruction to previous test items will 
presumably vary. Accordingly, we allowed teachers two positive responses when 
we asked if they taught the content of sampled items: "I teach this content," and "I 
teach similar content but not exactly this." In analysis, we initially kept the first 
response separate and then combined it with the second. Here we call the first 
"teach the same content" and the combined first and second "teach the same or 
similar content." Because of uncertainty about teachers' instructional responses to 
the test and about their characterization of their responses (just how similar does the 
content have to be before it is "this content" rather than "similar content?"), it is not 
clear a priori which of these two variables is likely to be more sensitive to 
reallocation. 

Considering first the "teach the same content" responses, eighth-grade teachers 
showed some degree of differentiation in their teaching emphasis between NAEP 
and MCAS items, which may be a sign of reallocation in response to the test. The 
mean proportions of items for which teachers reported teaching the same content 
was 56% for MCAS items and 30% for NAEP items. About 40% of the teachers said 
that they taught the content of none of the NAEP items, and almost three fourths 
said that they taught the content of either none or only one of them. This is shown in 
Figure 12, which is similar to the previous two figures: the X-axis indicates the 
proportion of items in response to which the teachers said they taught the same 
content, and each teacher is represented by two symbols, one for MCAS items and 
the other for NAEP items. The responses about MCAS items were more diverse. 
Only a single teacher reported teaching the same content as was reflected in all six of 
the MCAS items, but half of the teachers reported teaching the content of at least 
four of the six items, and three-fourths of the teachers reported teaching the content 
of half or more of the items. 

A similar differentiation of teaching appeared in the 10 th -grade. On average, 
teachers reported teaching the same content as was shown in 49% of the MCAS 
items and 26% of the NAEP items. However, 10 th -grade teachers were less likely 
than S^-grade teachers to report teaching the same content as was shown in most of 
the MCAS items (Figure 13). Only about one fourth of the teachers reported teaching 
the same content as was reflected in four or more of the six MCAS items. Here again, 
the small number of cases makes the meaningfulness of this difference between 
grades unclear. 












o 








O 


o 








r O 


o 


o 




o- 


o 


o 


o 




o- 


£ 


> 


o - 


h o 


— 


Q - 


- O 1 i 


* 


i o - 


-iO 




0.2 


0.4 




0.6 


0.8 


1.0 



0.0 

Proportion of items, same content tai 



Figure 12. Proportion of items for which each teacher reported 
she taught the same content. Grade 8 (+ symbol indicates NAEP 
items; o symbol indicates MCAS items). 
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Figure 13. Proportion of items for which each teacher reported 
she taught the same content, by teacher. Grade 10 (+ symbol 
indicates NAEP items; o symbol indicates MCAS items). 



Considering whether teachers teach the same or similar content, rather than 
just the same content, did not substantially change the differentiation between 
MCAS and NAEP items, but using this measure, differentiation appeared slightly 
weaker among 10 th -grade teachers. In the 8 th grade, the average teacher reported 
teaching the same or similar content for 78% of MCAS items and 48% of NAEP 
items. In the 10 th grade, the means were 69% for MCAS items and 50% for NAEP 
items. 
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Item-level results. In this section, the responses to teachers to the item-based 
questions are tabulated by item, rather than by teacher. For example, rather than 
calculating for each teacher the proportion of items correctly recognized as coming 
from MCAS, we calculated for each item the proportion of teachers who correctly 
identified it. We then plotted the distribution of these percentages across all items. 

Examining the responses by item showed more clearly the ability of many 
teachers to distinguish between MCAS and NAEP items. Figure 14 shows the 
percentage of teachers in both grades that responded that they recognized each 
sampled test item as coming from MCAS. In this and the following figures, each 
symbol represents a single item rather than a single teacher — in this case, the 
proportion of teachers reporting that they recognized that particular item as having 
come from MCAS. Therefore, each observation (i.e., each item) is given only one 
symbol, rather than two: a + symbol if the item was taken from NAEP, and a o 
symbol if the item was drawn from an earlier MCAS test. If teachers were perfectly 
accurate in identifying the source of items, all of the + symbols would be stacked at 
zero (all NAEP items would be "recognized" by no teachers as coming from the 
MCAS), while all of the o symbols would be stacked at 100 (all MCAS items would 
be correctly recognized by all teachers). Thus, for example, the right-most + symbol 
in this figure indicates that one NAEP item was incorrectly identified as an MCAS 
item by 24% of teachers. A few NAEP items were incorrectly identified by a modest 
percentage of teachers, and a number of MCAS items were not successfully 
identified by substantial percentages of teachers. Nonetheless, the two distributions 
are non-overlapping: all NAEP items were identified as coming from MCAS less 
often than the least frequently identified MCAS item. Contrary to our expectations, 
eighth-grade teachers were slightly more able to differentiate the two sets of items, 
but with so few teachers in the sample and so few items in the questionnaire, this 
may be a matter of chance. 
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Figure 14. Percent of teachers claiming to recognize each 
sample test item as coming from MCAS, both grades (+ symbol 
indicates NAEP items; o symbol indicates MCAS items). 



The items that were recognized by the largest proportions of teachers tended to 
be ones that the authors identified as being particularly characteristic of MCAS. At 
the 10 th grade, the cell phone billing item depicted in Figure 5 and the "puzzle" item 
depicted in Figure 6 were the most widely recognized (70% and 63% of teachers, 
respectively). Among Grade 8 teachers, the most frequently recognized items were a 
spatial-visualization task similar to that shown in Figure 3 (recognized by 71% of 
teachers), the geometric series item shown in figure 4 (62%), and the unconventional 
algebra problem shown in Figure 2 (62%). Thus it appears that teachers are aware of 
some of the features that characterize MCAS items. 

Tabulating responses by item is consistent with the earlier tabulations by 
teacher in providing some indication of reallocation in the eighth grade, but the 
evidence was weak in the 10 th grade. In the eighth grade, the content of most NAEP 
items was taught by only a small percentage of teachers, while the content of most 
MCAS items was taught by larger percentages of teachers (Figure 15). Some of the 
differences were small, however, and two items were out of order. In contrast, in the 
10 th grade, where one might expect stronger instructional responses because of 
higher stakes, the pattern was weaker: of the 10 items in the survey, only 2 MCAS 
items stood out from the other MCAS and NAEP items (see Figure 16). 
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Figure 15 . Percent of teachers teaching the same content 
as in each sample test item. Grade 8 (+ symbol indicates 
NAEP items; o symbol indicates MCAS items). 
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Figure 16. Percent of teachers teaching the same content 
as in sample test items, by item, Grade 10 (+ symbol indicates 
NAEP items; o symbol indicates MCAS items). 



When the percentage teaching the same or similar content was examined, the 
patterns changed, but differentiation between MCAS and NAEP items remained 
modest. In grade 8, three MCAS items were more often taught than other items; two 
NAEP items were taught less frequently; and two MCAS and one NAEP item were 
taught by roughly similar percentages of teachers (Figure 17). Again there was less 
differentiation in grade 10, where only two MCAS items and one NAEP item stood 
out from the others. 
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Figure 17. Percent of teachers teaching the same or similar 
content as in sample test items, grade 8 (+ symbol indicates 
NAEP items; o symbol indicates MCAS items). 

While the teacher-based and item-based displays showed roughly similar 
patterns in terms of differentiating teachers' responses to MCAS and NAEP items, 
the item-based display has the advantage of permitting analysis of specific test 
items. Variations between items in teachers' responses to a test may be central to 
evaluating the validity of gains. For example, the MCAS item the content of which 
the fewest eighth-grade teachers said they taught (Item J in Appendix) is an unusual 
and easily remembered item. It is a long algebra item with eight separate parts in 
which the unknowns (units of currency) are represented by unusual symbols 
(polygons with letters in them). Both authors considered this type of item to be 
characteristics of MCAS. Another item of this type, however, might employ a 
different, if also unusual, representation, so some teachers might characterize their 
preparation for this item as teaching "similar" content. In fact, about a fourth of 
eighth-grade teachers said they taught content similar to that in Item J. Yet when we 
examined the percentage of teachers teaching content the same as or similar to that 
in test items. Item J was still one of the MCAS items that received the least reported 
instructional focus (see Figure 17). 

Reasons for teaching content of items. For each test item, teachers who said 
they taught the content of the item were directed to a set of questions that asked the 
reasons for teaching it and whether their emphasis would change in the absence of 
MCAS. Teachers who said they did not teach the content were directed to a different 
set of questions that probed their reasons for not teaching it. The small numbers of 
respondents for these questions, and the fact that the numbers vary across test items 
depending on the frequency with which teachers reported teaching the content, 
makes these responses difficult to interpret. Nonetheless they suggest some possible 
evidence of reallocation and are worth examining, even though any conclusions 
must be quite tentative. We only examine the responses to the questions that 



teachers answered if they did report teaching the content, because the sample sizes 
for the others were prohibitively small. 

The first question asked teachers to select the reasons why they taught the 
content of the test item. Teachers were asked to select as many of the five reasons as 
applied. Table 12 summarizes the results for 8 th -grade teachers, and Table 13 
provides the same information for 10 th -grade teachers. The second column in Tables 
12 and 13 gives the percentage of teachers responding in the affirmative for each of 
the reasons, averaged across the six MCAS items. These numbers include only those 
teachers who reported teaching the content and who recognized the item from either 
MCAS or test-preparation materials. The third column gives the corresponding 
percentages for the four NAEP items and includes only those teachers who reported 
teaching the content and who either did not recognize the item or who recognized it 
from another test. 



Table 12 

Percentages of Teachers Who Reported Reasons for Teaching the Content of MCAS and 
NAEP Items, Grade 8 





MCAS Items 


NAEP Items 


It is part of the curriculum for this grade and course. 


52 


66 


It is not part of the curriculum for this grade and course, 
but I include it as review or to help students learn other 
material included in the curriculum. 


17 


34 


It helps students do well on MCAS. 


46 


20 


It helps students do well on other tests. 


14 


24 


It is preparation for more advanced mathematics courses. 


10 


12 



Table 13 

Percentages of Teachers Who Reported Reasons for Teaching the Content of MCAS and 
NAEP Items, Grade 10 





MCAS Items 


NAEP Items 


It is part of the curriculum for this grade and course. 


72 


85 


It is not part of the curriculum for this grade and course, 
but I include it as review or to help students learn other 
material included in the curriculum. 


11 


17 


It helps students do well on MCAS. 


48 


17 


It helps students do well on other tests. 


20 


21 


It is preparation for more advanced mathematics courses. 


25 


25 



In both grades, teachers were more likely to include "It helps students do well 
on MCAS" among their reasons for teaching MCAS items than they were for NAEP 
items, but some teachers did give this response for at least some of the NAEP items. 
Tenth-grade teachers were more likely to report teaching content than were 8 th - 
grade teachers because it is in the curriculum, which is consistent with some of the 
results reported earlier that suggested a stronger reliance on the curriculum among 
lO^-grade teachers. 

Another question asked teachers whether they made reference to MCAS when 
they taught the content of the test items. Tables 14 and 15 list the response options 
and the percentages of teachers who selected each, averaged over all MCAS items 
(second column) and over all NAEP items (third column). The MCAS column 
includes only teachers who reported teaching the content of the item and who 
recognized the item from MCAS or test-preparation materials. The NAEP column 
includes only those who reported teaching the content and who either did not 
recognize the item or who recognized it from another test. Table 14 provides results 
for S^-grade teachers, and Table 15 for lO^-grade teachers. 



Table 14 



Percentages of Teachers Who Said They Referred to MCAS While Teaching Item Content, 
Grade 8 





MCAS Items 


NAEP Items 


No reference 


18 


53 


Tell students this general content is likely to be included in 
MCAS 


58 


33 


Tell students an item like this one is likely to be included in 
MCAS 


24 


14 


Tell students this general content is not likely to be 
included in MCAS 


0 


0 


Tell students items like this one are not likely to be included 
in MCAS 


0 


0 




c. 4 0 
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Table 15 

Percentages of Teachers Who Said They Referred to MCAS While Teaching Item Content, 
Grade 10 





MCAS Items 


NAEP Items 


No reference 


19 


60 


Tell students this general content is likely to be included 
in MCAS 


45 


32 


Tell students an item like this one is likely to be included 
in MCAS 


34 


11 


Tell students this general content is not likely to be 
included in MCAS 


5 


0 


Tell students items like this one are not likely to be 
included in MCAS 


0 


0 



At both grades, teachers were more likely to report making no reference to 
MCAS when they taught the content of NAEP items than when they taught the 
content of MCAS items, providing additional evidence that teachers distinguish 
between material that is and is not represented in MCAS and that this distinction 
influences their instruction. However, teachers sometimes did refer to MCAS when 
presenting content from NAEP items, even though we selected NAEP items that did 
not resemble any items that had been included in MCAS tests. These results also 
indicate that teachers rarely explicitly told students that material would not be 
included in MCAS; instead, they simply did not present it. 

The final question in this section asked teachers whether they would change 
their emphasis on the content of the test item if students did not have to take MCAS. 
For present purposes, tabulations of these questions served two purposes. First, they 
provide a way of assessing reallocation without measuring it directly: Presumably 
teachers who say they would change their emphasis on a topic if MCAS were not a 
consideration are suggesting that MCAS has influenced their instructional choices 
with respect to that topic. Second, these tabulations provide an indication of 
whether item-based questions of this sort can successfully capture teachers' changes 
in emphasis in response to a high-stakes test. Tables 16 and 17 present the results, 
averaged across items, for the same groups of teachers included in Tables 15 and 16 
above. Table 16 shows fairly consistent but modest evidence of reallocation among 
8 th -grade teachers, with higher percentages reporting that they would decrease 
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Table 16 



Percentages of Teachers Who Said They Would Change Their 
Emphasis if Students Did Not Have to Take MCAS, Grade 8 





MCAS Items 


NAEP Items 


Decrease substantially 


22 


0 


Decrease slightly 


20 


14 


Would not change 


53 


82 


Increase slightly 


10 


14 


Increase substantially 


0 


0 



Table 17 

Percentages of Teachers Who Said They Would Change Their 
Emphasis if Students Did Not Have to Take MCAS, Grade 10 





MCAS Items 


NAEP Items 


Decrease substantially 


13 


0 


Decrease slightly 


12 


13 


Would not change 


72 


91 


Increase slightly 


4 


0 


Increase substantially 


0 


6 



emphasis on MCAS items than those who would on NAEP items. There was little 
evidence of this phenomenon in 10 th grade, a finding that is probably related to the 
greater curricular constraints on 10 th grade teachers. Together, the responses to these 
questions suggest some modest reallocation, particularly in the 8 th grade. More 
information would be needed to understand whether these findings generalize to a 
broader range of content and whether it would be possible to refine the questions to 
obtain better information on the nature of the reallocation. 

Inspection of responses to this set of questions for specific items illustrates 
some of the ways that teachers attend to specific types of test items. In the 10 th grade, 
there were twp items for which teachers chose the option, "It helps students do well 
on MCAS" as a reason for teaching the content of the item. One of them was Item A, 
depicted in Figure 6. More than two thirds of teachers reported that this item was 
helpful for MCAS, whereas only 44% said it was part of the curriculum — less than 
for any other MCAS or NAEP item on this survey. The other item was Item J, shown 
in Figure 18 below. This item asked students to determine which of four expressions 
yielded a negative result. Although nearly as many teachers (62%) said this item 
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Which of the following yields a negative result? 

A. (~5) 16 

B. (6 - 9)C4 + 2) 

C. -3(~4) 9 

D. ~2 9 

Figure 18. lOth-Grade MCAS Item J. 

helped students do well on MCAS as gave this response for Item A, Item J differed 
from Item A in the number who said it was part of the curriculum: 100 percent of 
responding teachers gave this response. This difference is evident in teachers' 
responses to the question about whether they would increase or decrease their 
emphasis in the absence of MCAS: 58% said they would decrease their emphasis 
either slightly or substantially on Item A, whereas only 8% gave these responses for 
Item J. Responses to the question about whether teachers make reference to MCAS 
when teaching the item were similar: 100% said they told students that the content 
of Item A or an item similar to Item A would be included in MCAS, and 93% gave 
these responses for Item J. It is clear from this discussion that teachers are aware that 
certain items are characteristic of MCAS, and that their decisions about whether to 
teach them are determined in part but not fully by the items' inclusion in MCAS. 

Relationships with general responses to MCAS. In this section we discuss 
descriptive analyses that we conducted to investigate whether teachers' responses to 
the item-based questions showed the expected relationships with one another and 
with some of the other information collected through the surveys. We have 
insufficient data to investigate these relationships in detail, and as we discussed 
earlier, none of our results should be interpreted as indicative of relationships that 
might be found in a larger or more representative sample of teachers. These analyses 
are purely exploratory, and as with other material presented here, are intended to 
provide guidance for future data collection efforts. 

Table 18 presents correlations among the proportions of MCAS and NAEP 
items recognized and taught, with correlations for S^-grade teachers above the main 
diagonal and those for lO^-grade teachers below it. We hypothesized that teachers 
who were more accurate at recognizing MCAS items would be less likely than other 
teacher to incorrectly attribute NAEP items to MCAS, which would produce a 
negative correlation between "MCAS recognize" and "NAEP recognize." We did 
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Table 18 



Correlations Among Proportions of Items Recognized and Taught (Grade 8 Above 
Diagonal and Grade 10 Below Diagonal) 





MCAS recognize 


NAEP recognize 


MCAS teach 


NAEP teach 


MCAS recognize 


1.00 


.20 


.19 


-.02 


NAEP recognize 


.20 


1.00 


-.06 


-.31 


MCAS teach this 


.22 


.12 


1.00 


.39 


NAEP teach this 


-.10 


-.08 


.27 


1.00 



not see evidence of this expected relationship; on the contrary, the correlations were 
positive but very small (.20) in both grades. Similarly, we wanted to explore whether 
the proportion of MCAS items taught was negatively related to the proportion of 
NAEP items taught. Again, the correlations were positive: r=A in the 8 th grade and 
r=.3 in the 10 th grade. These unexpected positive relationships could stem from any 
of several causes, and larger studies would be needed to disentangle them. For 
example, the modest positive correlations between reporting that one teaches MCAS 
items and NAEP items could reflect teaching styles (some teachers may cover a 
broader range of content than others and therefore tend to answer "yes" to more 
item-based questions) or response bias, as well as simple sampling error resulting 
from our very small samples. 

Next we examined relationships between the item-based questions and 
teachers' reliance on test-preparation activities. Positive relationships between these 
sets of questions would provide support for the validity of teachers' responses, in 
that concern with tailoring instruction to the MCAS could lead teachers both to 
attend to earlier MCAS items and to use MCAS-related test-preparation materials. 
Table 19 presents results for 8 th grade, and Table 20 provides the 10 th -grade results. 
We examined fewer test-preparation resources in 8th than in 10 th grade because 
many of the resources were used by very few 8 th -grade teachers (see Tables 5 and 6). 
At both grades, use of test-preparation materials tended to be positively correlated 
with proportion of MCAS items recognized, and nearly uncorrelated with 
proportion of NAEP items recognized. An exception to the latter finding was the 
correlation of .36 between NAEP items recognized and use of other (i.e., non-MCAS) 
commercial test-preparation materials at 10 th grade. This is not surprising, as the use 
of materials not developed for MCAS probably provides teachers with exposure to a 
broader range of test item types than they would get from MCAS-specific materials, 
but it suggests that teachers who use non-MCAS materials may not be aware of the 



Table 19 



Spearman Correlations Between Item-Level Responses and Use of Test-Preparation 
Resources, Grade 8 





Use of old MCAS 
items 


Use of other sample 
MCAS items 


Use of MCAS 
scoring rubrics 


MCAS recognize 


.57 


.45 


.44 


NAEP recognize 


.06 


-.10 


.01 


MCAS teach this 


.33 


.32 


.26 


NAEP teach this 


.21 


.05 


-.07 



Table 20 



Spearman Correlations Between Item-Level Responses and Use of Test-Preparation Resources, 
Grade 10 





Use of old 
MCAS items 


Use of other 
sample MCAS 
items 


Use of MCAS 
scoring rubrics 


Princeton 
Review's 
Cracking the 
MCAS 


Other 

commercial test- 
prep materials 


MCAS recognize 


.53 


.20 


-.03 


.19 


.12 


NAEP recognize 


.13 


.14 


-.03 


.02 


.36 


MCAS teach this 


.16 


.24 


.12 


.08 


.05 


NAEP teach this 


.02 


-.12 


-.06 


-.24 


-.01 



differences between MCAS items and other test items. Relationships with the 
proportions of items taught were mixed, but generally positive for MCAS and 
negative or close to zero for NAEP. 

Finally, we examined correlations between the item-based questions and 
teachers' responses to the question regarding ways in which their teaching is 
affected by MCAS (see Tables 8 and 9 above for summaries of responses to this 
question). In this case as well, relationships among responses to these questions 
could provide convergent evidence of the validity of teachers' responses. However, 
the relationships are inconsistent between the two grade levels (see Tables 21 and 
22) and generally do not provide convergent evidence of validity. Larger samples 
would be needed to ascertain whether these patterns reflect sampling error or 
limitations of these forms of questions. One consistent pattern that warrants 
additional exploration is that teachers who reported focusing on the Massachusetts 
content standards did not show any tendency to recognize more MCAS items or to 
report teaching the content of MCAS items. In addition, in both grades, these 



ERIC 



43 . 



45 



Table 21 



Spearman Correlations Between Item-Level Responses and Changes to Teaching, Grade 8 





Focus more on 
MA Content 
Standards (c) 


Shift among 
topics of math 
curriculum to 
focus on material 
in MCAS (d) 


Emphasize 
writing with 
mathematics (1) 


Look for problem Spend more time 
styles and on math test- 

formats in MCAS taking strategies 
(g) (i) 


MCAS recognize 


.08 


.31 


.39 


.16 


.43 


NAEP recognize 


.39 


.33 


-.08 


.20 


.25 


MCAS teach this 


-.09 


.42 


.16 


.15 


.19 


NAEP teach this 


.15 


.07 


.02 


.22 


.03 


Table 22 












Spearman Correlations Between Item-Level Responses and Changes to Teaching, Grade 10 




Focus more on 
MA Content 
Standards (c) 


Shift among 
topics of math 
curriculum to 
focus on material 
in MCAS (d) 


Emphasize 
writing with 
mathematics (1) 


Look for 
problem styles 
and formats in 
MCAS (g) 


Spend more 
time on math 
test-taking 
strategies (i) 


MCAS recognize 


-.20 


.02 


-.03 


-.08 


.08 


NAEP recognize 


.42 


.08 


.26 


.27 


.58 


MCAS teach this 


.08 


-.02 


-.08 


.11 


.42 


NAEP teach this 


.01 


-.29 


-.18 


-.21 


.18 



teachers were more likely to label NAEP items incorrectly as coming from MCAS. 
This raises the possibility that in the behavior of some teachers, focusing on 
standards and focusing on the test used to implement them may be distinct and 
independent. 

Content-Based Questions 

The final section of the survey was similar to the third, but instead of 
presenting test items, we selected seven specific content categories that were 
classified as having either high or low representation on MCAS. 

In contrast to the responses to the questions about test items, responses to 
questions about content areas differentiated only inconsistently between high- and 
low-MCAS categories. In the eighth grade, the large majority of teachers reported 
teaching the content of 4 of the 5 content areas that we found stressed in the MCAS 
forms we examined (see Table 23). However, only about half reported teaching the 
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content of the fifth area stressed in MCAS (transformations), and 55% and 86% 
reported teaching the content of the two NAEP content areas that we found not to be 
stressed in the MCAS (proportional thinking and systems of equations). In the 10 th 
grade, the large majority of teachers reported teaching the content of two of our 
content areas stressed in the MCAS, but two others were essentially no different 
from two of the areas not stressed in the MCAS (see Table 24). We return to these 
ambiguous results in the Discussion section below. 



Table 23 



Percentages of Teachers Reporting That They Teach Specific Content Categories, Grade 8 
(N in Parentheses) 





No, I don't 
teach this 
content 


I teach this 
content 


I teach similar 
content but not 
exactly this 


I teach this 
content and teach 
similar content 


High-MCAS 


B (percentages) 


0 


95 (20) 


0 


5(1) 


C (perimeter and area) 


0 


95 (20) 


0 


5(1) 


D (transformations) 


29 (6) 


48 (10) 


14 (3) 


10 (2) 


E (measuring angles) 


0 


90 (19) 


5(1) 


5(1) 


F (extending patterns) 


5(1) 


85 (17) 


5(1) 


5(1) 


Low-MCAS 


A (proportional thinking) 


5(1) 


86 (18) 


10(2) 


0 


G (systems of equations) 


30 (6) 


55 (11) 


5(1) 


10 (2) 



Table 24 

Percentages of Teachers Reporting That They Teach Specific Content Categories, Grade 10 
(N in Parentheses) 





No, I don't 
teach this 
content 


I teach this 
content 


I teach similar I teach this 

content but not content and teach 
exactly this similar content 


High-MCAS 


A (transformations) 


27 (8) 


47 (14) 


23(7) 


3(1) 


C (measuring angles) 


7(2) 


80 (24) 


10 (3) 


3(1) 


D (extending patterns) 


7(2) 


83 (25) 


3(1) 


7(2) 


F (factoring equations) 


60 (18) 


33 (10) 


7(2) 


0 


Low-MCAS 


B (proportional thinking) 


3(1) 


39 (28) 


3(1) 


0 


E (basic trigonometry) 


66 (19) 


34 (10) 


0 


0 


G (complex figures) 


20(6) 


67 (20) 


10 (3) 


3(1) 
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Discussion 



This study was undertaken to explore whether surveys can be made useful for 
identifying the types of responses to high-stakes testing that Koretz et al. (2001) 
maintained are a key to evaluating the validity of score gains. 

Simply describing in survey questions the categories of responses outlined by 
Koretz et al. (2001) appears to be inadequate. We found it extremely difficult to 
describe the relevant categories of responses clearly in the context of short survey 
questions, and our interviews with respondents suggested that many teachers could 
not clearly understand the distinctions we were trying to make. 

Survey questions based on the content areas specified in test frameworks might 
in theory function better than these general question because they add specificity, 
but our questions based on content areas also appeared to be relatively ineffective. 
Simple verbal descriptions of content areas did not enable teachers in our sample to 
differentiate consistently between the areas emphasized on the test and those given 
less emphasis. Questions based on content areas would also be limited in application 
even if they were more effective because they cannot be used to pinpoint aspects of 
item style or non-substantive elements that could be leverage points for coaching. 

Questions based on actual test items seem to be the most promising of the 
approaches tried here. Most teachers reported placing moderate to great emphasis 
on old test items in preparing students for the test, and in responses to our surveys, 
they showed a moderately good ability to differentiate between MCAS items and 
NAEP items. Several relationships between responses to item-based questions and 
other questions offer support for the validity of the item-based questions. Moreover, 
item-based questions can be used to explore a variety of aspects of a high-stakes test, 
including both content and many aspects of item style, such as format of 
presentation. Accordingly, these questions may be useful for exploring various 
forms of coaching as well as reallocation. 

A potentially major limitation of all of the approaches explored here is that 
they are not designed to address instructional decisions made by people other than 
classroom teachers. The questions explored in this study were predicated on the 
assumption that teachers make decisions about how to align instruction with the 
high-stakes test and therefore should be able to describe the aspects of the test on 
which they focus their instruction. We were surprised by the frequency with which 
the teachers in our sample reported that they had little flexibility to modify their 
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instruction in response to the high-stakes test. To the extent that other parties, such 
as district administrators, make key decisions about alignment, it will be necessary 
to modify the methods explored here and supplement them with others designed to 
capture the decisions made by other parties. 
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BOSTON PUBLIC SCHOOLS, GRADE 8 
STUDY BACKGROUND 



This questionnaire and the associated telephone interview are part of an ongoing effort to strengthen the 
evaluation of high-stakes testing programs. 

NOTE: This survey is shorter than it appears. It includes considerable blank space (because of formatting), 
and numerous questions that you will be asked to skip (depending on your answers to other questions). In 
several cases, we ask you one set of questions repeatedly, with each repetition focused on a specific test item or 
content area. After the first few repetitions, you will find these sections easier to fill out quickly. Teachers who 
pilot-tested this survey completed it in 40 to 60 minutes. 

Although your district has approved the study, the study is not being conducted for the district or for the state 
department of education. This is an independent study conducted by researchers at the Harvard Graduate 
School of Education and RAND, in their role as partners in the Center for Research on Evaluation, Standards, and 
Student Testing (CRESST). 

This study is not an evaluation of teachers or schools. We will not share your responses with anyone, including 
school or district personnel. We will ask for your name, but this is solely for logistical purposes, such as linking 
this form to your interview and mailing you your honorarium check. Identifying information will be destroyed 
once those needs are met. 

This is a pilot study designed to test and evaluate methods of obtaining information from teachers about their 
responses to high-stakes testing programs. To the extent that these methods are successful, we expect to use them 
in future, large-scale studies of high-stakes testing programs. 

We will ask you a variety of questions about your responses to the MCAS testing program. We will also ask you 
to evaluate some aspects of our survey questions and provide feedback to help us improve them. 

Please keep this questionnaire after you have filled it in as a reference for your interview. When you have 
completed the interview, please mail this questionnaire back to us in the attached stamped envelope. When you 
have completed both this questionnaire and the telephone interview, we will mail you an honorarium of $100 as a 
gesture of appreciation for your time and effort. 

Substantive questions can be sent to the study directors, Daniel Koretz (daniel koretz@harvard.edu , 617-384- 
8090) or Laura Hamilton ( Laura Hamilton@rand.org, 310-393-0411 x 6146). Because of travel schedules, we 
strongly urge you to use e-mail if possible. 

Thank you very much for taking the time to share your expertise with us. 



BEST COPY AVAILABLE 
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General instructions: 



This survey includes three types of questions: 

• A few questions ask you to write in an answer. 

• A few questions ask you to choose all appropriate choices from a list. All of these questions specifically 
note that in their directions. 

• All other questions should be answered by selecting the single best answer from among the available 
choices. 

You should indicate your answer(s) by checking the box by the choice or by circling the number where there is no 
box. 

In some cases, you may find that the phrasing of a question or the choice of responses is not optimal to describe your 
responses to MCAS. Please make notes about these cases as you fill out the survey. During your phone interview, 
you will be asked to note any questions that you found particularly problematic. Your feedback about these 
questions will be used to revise this survey for future use in evaluating high-stakes testing programs. 

background information 

This information will be used only to complete data collection and will be destroyed once interviews 
have been linked to this survey instrument. No identifying information will be retained or displayed. 



Name: 

District: 

Preferred phone number for interview: 

Alternative phone number for interview: 

Address for mailing survey forms and honorarium check: 



E-mail address: (will be used only for scheduling, etc.) 

Social security number 

Your SSN is required for us to pay your honorarium and will not be used for any other purpose. 

1. How many periods of mathematics do you teach on a typical day? 

2. Please list the title(s) of the classes you teach (e.g., algebra, pre-algebra). 
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3. If you teach eighth-grade mathematics to more than one class, please respond to this questionnaire basedon 
your experiences in the first eighth-grade mathematics class of the day . We refer to this as your "target" class. 
What type of course is your target class? (e.g., algebra, pre-algebra, etc.) 

4. How does the math achievement of this school compare to that of other schools in the state? 

a 1. Substantially below average 
a 2. Moderately below average 
a 3. About average 
a 4. Moderately above average 
a 5. Substantially above average 

5. How does the math achievement of the class about which you will respond compare to that of other classes m 
this school ? 

a 1. Substantially below average 
a 2. Moderately below average 
a 3. About average 
a 4. Moderately above average 
a 5. Substantially above average 

general responses to MCAS 

1. How much pressure do you feel to improve your students' scores on the MCAS Mathematics test? 

a l.None 
a 2. A small amount 
a 3. A moderate amount 
° 4. A great deal 

2. How well is MCAS aligned with the curriculum for this course in your school? 

a 1. Poorly, because MCAS includes considerable material that is not in our curriculum 
a 2. Poorly, because MCAS omits considerable material that is in our curriculum. 
a 3. Fairly well, although the overlap is not complete. 
a 4. Very well. 
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3. How much emphasis do you place on each of the following resources to help students do well on the MCAS 
Mathematics test? 



a. Old MCAS items 

b. Other sample MCAS test items 

c. MCAS scoring rubrics 

d. The Princeton Review's Cracking the MCAS 

e. The MCAS2003.com web site 

f. The BPS "Tool Kit" 

g. Other commercial test-prep books or materials 

h. Other DOE or BPS test-preparation materials 



No 


Slight 


Moderate 


Great 


emphasis 


emphasis 


emphasis 


emphasis 


1 


2 


3 


4 


1 


2 


3 


4 


1 


2 


3 


4 


1 


2 


3 


4 


1 


2 


3 


4 


1 


2 


3 


4 


1 


2 


3 


4 


1 


2 


3 


4 



4. If you give moderate or great emphasis to any test preparation materials other than those listed in 2(a) through 
2(f) in the preceding question, please list the materials you use most. 



5. How much flexibility do you have to modify your curriculum and instruction in response to MCAs? 
a 1 • Very little because my school or district requires me to adopt a specific curriculum. 
a 2. Very little, but for some other reason. Specify: 

a 3. A moderate amount. 

^ 4. A great deal. 
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6. Please think about ways in which your teaching is different because of MCAS than it would be without MCAS. 
Indicate the extent to which any of the following statements characterizes those differences. In your phone 
interview, we will ask you for additional detail about some of these questions. 



Not at all 

a. I assign more homework or more difficult homework. 1 

b. MCAS has prompted me to find more effective teaching methods ^ 

(more effective in general, not specifically targeted to MCAS). 

c. I focus more on the Massachusetts Curriculum Framework ^ 

Content Standards in mathematics. 

d. I shift emphasis among topics or strands of the mathematics 

curriculum to focus more on material emphasized in MCAS (e.g., ^ 

shifting instructional time from coordinate geometry to plane 
geometry or vice-versa). 

e. I shift emphasis from other mathematics material not in the local ^ 

curriculum to focus more on material emphasized in MCAS. 

f. I look for patterns in the ways that MCAS presents a given type 

of content and emphasize those in my instruction, (e.g., using 1 

coordinate geometry to present algebra problems) 

g. I look for particular styles and formats of problems in MCAS and 

emphasize those in my instruction, (e.g., using unusual pictures ^ 

or symbols to present algebra problems; using specific key 

phrases) 

h. I focus more on the aspects of performance that are stressed by ^ 

the MCAS scoring rubrics. 

i. I place more stress than I would on mathematics test-taking 
strategies, such as "plugging in" and "process of elimination." 

j. I spend more time teaching and encouraging generic test-taking ^ 

strategies such as time management or eating a good breakfast. 

k. I spend more time teaching mathematics content (e.g., by 

replacing non-ins tructional activities with instruction on 1 

mathematics). 

l. I place a greater emphasis on writing with mathematics. 1 



Somewhat 

2 

2 

2 

2 

2 

2 

2 



great deal 

3 

3 

3 

3 

3 

3 

3 



2 3 



2 3 

2 3 



2 3 

2 3 



7. TO WHAT EXTENT DID YOUR SCHOOL’S MCAS SCORES RISE BETWEEN SPRING 2000 AND SPRING 2001? 
a 1. Not at all 
a 2. A small amount 
a 3. A moderate amount 
a 4. A great deal 
a 5. Don't know 



IF “NOT AT ALL” OR “DON’T KNOW”, SKIP THE NEXT ITEM. 



BEST COPY AVAILABLE 



8. Below are some factors which may have contributed to the score gain that occurred in your school. For each, 
please circle the number indicating the extent to which you think it contributed to score improvements. Please 
circle only one number in each row. 







Not at all 


A small 
amount 


A 

moderate 

amount 


A great 
deal 


Don't 

know 


a. 


Improved test-taking skills 


1 


2 


3 


4 


9 


b. 


Increased student motivation 


1 


2 


3 


4 


9 


c. 


Differences between student cohorts 
from year to year 


i 


2 


3 


4 


9 


d. 


Broad improvements in students' 
knowledge and skills 


i 


2 


3 


4 


9 


e. 


Improvements in students' mastery 
of knowledge and skills that are 
emphasized in MCAS 


i 


2 


3 


4 


9 


f. 


Increased familiarity with the MCAS 
assessments 


i 


2 


3 


4 


9 


g- 


Work with practice tests and other 
preparation materials 


i 


2 


3 


4 


9 
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questions about test items 



In this section, we will ask you a series of questions about several mathematics test items. You will be asked 
to skip some questions based on your response to the first question following each item. The arrows will 
show you where to go. 

NOTE: When we use the term "content," it refers to the knowledge AND the skills needed to answer a given 
test item. 

Item A . lmelda will work 10 to 20 hours per week at her new job and will be paid $7.50 per hour. Which of the 
following shows how much she can earn per week? 

A I I I I 1 >,i I 1 1 0 I II > 

$0 $30 $60 $90 $120 $150 $180 

R l ll l l ft l l ll O I M > 

a ■ $0 $30 $60 $90 $120 $150 $180 

r I II I I » I I 1 1 f I I I » 

$0 $30 $60 $90 $120 $150 $180 

D I I I M + I I I ! ♦ I I I ► 

■ $0 $30 $60 $90 $120 $150 $180 

Al. Do you recognize this test item? 

□ 1. No 

a 2. Yes, from MCAS 
a 3. Yes, from another test 

Which test? 

O 4. Yes, from test-preparation materials, a textbook, or other instructional materials 

Which materials? 

a 5. Yes, but I'm not sure where Fve seen it. 

A2. What must a student know or be able to do in order to answer this item correctly? Please briefly note the 
skill or knowledge you consider most essential, (e.g., "must be able to do simple computations with exponents/') 
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A3. Do you teach content of this test item in this class? 



a 1. No, I don't teach this content. Continue to question A4 . 

a 2. 1 teach this content. Skip to question A6 p 

° 3. I teach similar content but not exactly this (e.g., "I teach area, but only with regular polygons"). 

Skip to question A6 . ^ 

If you answered “no” to question A3, continue to question A4. Otherwise, skip to question A 6. 



A4. Why do you not teach this content? 

a 1. It is not in the curriculum. 

a 2. It is taught in an earlier grade. 

a 3. It is taught in a later grade. 

° 4. It is in the curriculum, but it is not an 

important topic. 

a 5. It is in the curriculum but it is not 
emphasized or included in MCAS. 

a 6. Other reason (please specify) 



A6. For which of the following reasons do ymi teach 
the content of this test item? (Select as many as 
apply.) 



a 1. It is part of the curriculum for this grade 
and course 



° 2. It is not part of the curriculum for this 

grade and course, but I include it as review or to 
help students learn other material included in 
the curriculum. 

a 3. It helps students do well on MCAS. 
a 4. It helps students do well on other tests. 



° 5. It is preparation for more advanced 

mathematics courses 



A5. Would you change your emphasis on the 
content of this item if your students did not have to 
take MCAS? 

a 1. 1 still would not teach it. 

° 2. 1 would give it slight emphasis 

a 3. 1 would give it substantial emphasis 
Go to Item B. 



A7. How much emphasis do you give to the content 
of this test item? 

a 1. It is taught in one lesson; not the primary 
focus of that lesson. 

° 2. It is the primary or sole focus of a single 

lesson. 

° 3. It is the primary or sole focus of two 

lessons. 

a 4. It is taught over an extended period of 
time (e.g., woven through an investigation). 

A8. Which of the following best characterizes the 
way you teach this content? 

° 1. 1 have presented this test item 

° 2. 1 present problems with similar content 

and similar format 

° 3. 1 present problems with similar content 

that do not resemble this test item 

a 4. 1 present this content in the context of 
other topics but do not present problems that 
focus primarily on this content 



▼ 
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from question A5 



Go to Item B. 



t 



A9. Do you make reference to MCAS when you 
teach the content of this item? 

□ 1. No 

a 2. Yes; I tell students that this general 
content is likely to be included in MCAS. 

a 3. Yes; I tell students that an item like this 
one is likely to be included in MCAS. 

a 4. Yes; I tell students that this general 
content is not likely to be included in MCAS. 

a 5. Yes; I tell students that items like this are 
not likely to be included in MCAS. 

A10. Would your emphasis on the content of this 
item change if your students did not have to take 
MCAS? 

O 1. It would decrease substantially 
O 2. It would decrease slightly 

□ 3. It would not change 

□ 4. It would increase slightly 

O 5. It would increase substantially 
Go to Item B 



Item B . Eva has four sets of straws. The measurements of the straws are given below. Which set of 
straws could not be used to form a triangle? 

A. Set 1: 4 cm, 4 cm, 7 cm 

B. Set 2: 2 cm, 3 cm, 8 cm 

C. Set 3: 3 cm, 4 cm, 5 cm 

D. Set 4: 5 cm, 12 cm, 13 cm 

Bl. Do you recognize this test item? 
a 1. No 

a 2. Yes, from MCAS 
a 3. Yes, from another test 

Which test? 

a 4. Yes, from test-preparation materials, a textbook, or other instructional materials 

Which materials? 

B2. What must a student know or be able to do in order to answer this item correctly? Please briefly note the 
skill or knowledge you consider most essential, (e.g., '"must be able to do simple computations with 
exponents.") 
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B3. Do you teach content of this test item in this class? 

a 1. No, I don't teach this content. Continue to question B4 . 

a 2. 1 teach this content. Skip to question B6 w 

a 3. I teach similar content but not exactly this (e.g., "I teach area, but only with regular polygons"). 
Skip to question B6 . ^ 

If you answered “no” to question B3, continue to question B4. Otherwise, skip to question B6. 



B4. Why do you not teach this content? 

a 1. It is not in the curriculum. 

a 2. It is taught in an earlier grade. 

a 3. It is taught in a later grade. 

a 4. It is in the curriculum, but it is not an 
important topic. 

a 5. It is in the curriculum but it is not 
emphasized or included in MCAS. 

a 6. Other reason (please specify) 



B5. Would you change your emphasis on the 
content of this item if your students did not have to 
take MCAS? 

a 1. 1 still would not teach it. 
a 2. 1 would give it slight emphasis 
a 3. 1 would give it substantial emphasis 
Go to Item C. 



B6. For which of the following reasons do you teach 
the content of this test item? (Select as many as 
apply.) 

a 1. It is part of the curriculum for this grade 
and course 

a 2. It is not part of the curriculum for this 
grade and course, but I include it as review or to 
help students learn other material included in 
the curriculum. 

a 3. It helps students do well on MCAS. 

° 4. It helps students do well on other tests. 

a 5. It is preparation for more advanced 
mathematics courses 

B7. How much emphasis do you give to the content 
of this test item? 

a 1 . It is taught in one lesson; not the primary 
focus of that lesson. 

a 2. It is the primary or sole focus of a single 
lesson. 

a 3. It is the primary or sole focus of two 
lessons. 

a 4. It is taught over an extended period of 
time (e.g., woven through an investigation). 

B8. Which of the following best characterizes the 
way you teach this content? 

a 1. 1 have presented this test item 

a 2. 1 present problems with similar content 
and similar format 

a 3. 1 present problems with similar content 
that do not resemble this test item 

a 4. 1 present this content in the context of 
other topics but do not present problems that 
focus primarily on this content 
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from question B5 



▼ 

Go to Item C 



B9. Do you make reference to MC AS when you 
teach the content of this item? 

□ 1. No 

□ 2. Yes; I tell students that this general 
content is likely to be included in MCAS. 

□ 3. Yes; I tell students that an item like this 
one is likely to be included in MCAS. 

□ 4. Yes; I tell students that this general 
content is not likely to be included in MCAS. 

□ 5. Yes; I tell students that items like this are 
not likely to be included in MCAS. 

BIO. Would your emphasis on the content of this 
item change if your students did not have to take 
MCAS? 

a 1. It would decrease substantially 
a 2. It would decrease slightly 
a 3. It would not change 
a 4. It would increase slightly 

□ 5. It would increase substantially 
Go to Item C. 



Item C . A poll is being taken at Baker Junior High School to determine whether to change the school 
mascot. Which of the following would be the best place to find a sample of students to 
interview that would be most representative of the entire student body? 

A) An algebra class 

B) The cafeteria 

C) The guidance office 

D) A French class 

E) The faculty room 

Cl. Do you recognize this test item? 

□. 1. No 

a 2. Yes, from MCAS 
a 3. Yes, from another test 

Which test? 

a 4. Yes, from test-preparation materials, a textbook, or other instructional materials 

Which materials? 

C2. What must a student know or be able to do in order to answer this item correctly? Please briefly note the 
skill or knowledge you consider most essential, (e.g., "must be able to do simple computations with 
exponents.") 
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C3. Do you teach content of this test item in this class? 

a 1. No, I don't teach this content. Continue to question C4 . 

a 2. 1 teach this content. Skip to question C6 

a 3. I teach similar content but not exactly this (e.g., "I teach area, but only with regular polygons"). 
Skip to question C6 . 

If you answered “no” to question C3, continue to question C4. Otherwise, skip to question C6. 



C4. Why do you not teach this content? 

a 1. It is not in the curriculum. 

a 2. It is taught in an earlier grade. 

a 3. It is taught in a later grade. 

a 4. It is in the curriculum, but it is not an 
important topic. 

a 5. It is in the curriculum but it is not 
emphasized or included in MCAS. 

a 6. Other reason (please specify) 



C5. Would you change your emphasis on the 
content of this item if your students did not have to 
take MCAS? 

a 1. 1 still would not teach it. 
a 2. 1 would give it slight emphasis 
a 3. 1 would give it substantial emphasis 
Go to Item D. 



C6. For which of the following reasons do you teach 
the content of this test item? (Select as many as 
apply.) 

a 1. It is part of the curriculum for this grade 
and course 

a 2. It is not part of the curriculum for this 
grade and course, but I include it as review or to 
help students learn other material included in 
the curriculum. 

a 3. It helps students do well on MCAS. 

a 4. It helps students do well on other tests. 

a 5. It is preparation for more advanced 
mathematics courses 

C7. How much emphasis do you give to the content 
of this test item? 

O 1. It is taught in one lesson; not the primary 
focus of that lesson. 

a 2. It is the primary or sole focus of a single 
lesson. 

a 3. It is the primary or sole focus of two 
lessons. 

a 4. It is taught over an extended period of 
time (e.g., woven through an investigation). 

C8. Which of the following best characterizes the 
way you teach this content? 

a 1. 1 have presented this test item 

a 2. 1 present problems with similar content 
and similar format 

a 3. 1 present problems with similar content 
that do not resemble this test item 

a 4. 1 present this content in the context of 
other topics but do not present problems that 
focus primarily on this content 



▼ 
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from question C 5 



Go to Item D. 



▼ 



C9. Do you make reference to MCAS when you 
teach the content of this item? 

□ 1. No 

□ 2. Yes; I tell students that this general 
content is likely to be included in MCAS. 

□ 3. Yes; I tell students that an item like this 
one is likely to be included in MCAS. 

□ 4. Yes; I tell students that this general 
content is not likely to be included in MCAS. 

□ 5. Yes; I tell students that items like this are 
not likely to be included in MCAS. 

CIO. Would your emphasis on the content of this 
item change if your students did not have to take 
MCAS? 

□ 1. It would decrease substantially 

□ 2. It would decrease slightly 
a 3. It would not change 

□ 4. It would increase slightly 

O 5. It would increase substantially 

Go to Item D. 



Item D . Each arrangement in this pattern is made up of tiles. 




How many tiles will be in the 6 th arrangement in the pattern? 

Dl. Do you recognize this test item? 

□ 1. No 

a 2. Yes, from MCAS 
a 3. Yes, from another test 

Which test? 

° 4. Yes, from test-preparation materials, a textbook, or other instructional materials 



Which materials? 



D2. What must a student know or be able to do in order to answer this item correctly? Please briefly note the 
skill or knowledge you consider most essential, (e.g., "must be able to do simple computations with exponents.") 
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D3. Do you teach content of this test item in this class? 

a 1. No, I don't teach this content. Continue to question D4 . 

a 2. 1 teach this content. Skip to question D6 . ^ 

a 3. I teach similar content but not exactly this (e.g., "I teach area, but only with regular polygons"). 
Skip to question D6 . ^ 

If you answered “no” to question D3, continue to question D4. Otherwise, skip to question D6. 



. D4. Why do you not teach this content? 

a 1. It is not in the curriculum. 

a 2. It is taught in an earlier grade. 

a 3. It is taught in a later grade. 

a 4. It is in the curriculum, but it is not an 
important topic. 

a 5. It is in the curriculum but it is not 
emphasized or included in MCAS. 

a 6. Other reason (please specify) 



D5. Would you change your emphasis on the 
content of this item if your students did not have to 
take MCAS? 

a 1. 1 still would not teach it. 
a 2. 1 would give it slight emphasis 
a 3. 1 would give it substantial emphasis 

Go to item E . 



D6. For which of the following reasons do you teach 
the content of this test item? (Select as many as 
apply.) 

□ 1. It is part of the curriculum for this grade 
and course 

a 2. It is not part of the curriculum for this 
grade and course, but I include it as review or to 
help students learn other material included in 
the curriculum. 

a 3. It helps students do well on MCAS. 

□ 4. It helps students do well on other tests. 

a 5. It is preparation for more advanced 
mathematics courses 

D7. How much emphasis do you give to the content 
of this test item? 

a 1. It is taught in one lesson; not the primary 
focus of that lesson. 

O 2. It is the primary or sole focus of a single 
lesson. 

° 3. It is the primary or sole focus of two 

lessons. 

a 4. It is taught over an extended period of 
time (e.g., woven through an investigation). 

D8. Which of the following best characterizes the 
way you teach this content? 

a 1. 1 have presented this test item 

. O 2. 1 present problems with similar content 
and similar format 

a 3. 1 present problems with similar content 
that do not resemble this test item 

O 4. 1 present this content in the context of 
other topics but do not present problems that 
focus primarily on this content 



▼ 
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from question D 5 



Go to Item E 



▼ 



D9. Do you make reference to MCAS when you 
teach the content of this item? 

□ 1. No 

□ 2. Yes; I tell students that this general 
content is likely to be included in MCAS. 

a 3. Yes; I tell students that an item like this 
one is likely to be included in MCAS. 

a 4. Yes; I tell students that this general 
content is not likely to be included in MCAS. 

a 5. Yes; I tell students that items like this are 
not likely to be included in MCAS. 

DIO. Would your emphasis on the content of this 
item change if your students did not have to take 
MCAS? 

a 1. It would decrease substantially 
a 2. It would decrease slightly 
a 3. It would not change 
a 4. It would increase slightly 
a 5. It would increase substantially 
Go to Item E. 



Item E . Use the unit of length below to estimate the perimeter of the figure shown. Between which two 
consecutive whole-number units does the perimeter lie? 




Answer: Between. 



and . 



El. Do you recognize this test item? 
□ 1. No 

° 2. Yes, from MCAS 

a 3. Yes, from another test 
Which test? 



4. Yes, from test-preparation materials, a textbook, or other instructional materials 
Which materials? 
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E2.What must a student know or be able to do in order to answer this item correctly? Please briefly note the skill 
or knowledge you consider most essential, (e.g., "must be able to do simple computations with exponents.") 



E3. Do you teach content of this test item in this class? 

a 1. No, I don't teach this content. Continue to question E4 . 

a 2. 1 teach this content. Skip to question E& ► 

a 3. I teach similar content but not exactly this (e.g., "I teach area, but only with regular polygons"). 
Skip to question E6 . ^ 

If you answered “no” to question E3, continue to question E4. Otherwise, skip to question E6. 



E4. Why do you not teach this content? 

a 1. It is not in the curriculum. 

° 2. It is taught in an earlier grade. 

a 3. It is taught in a later grade. 

a 4. It is in the curriculum, but it is not an 
important topic. 

a 5. It is in the curriculum but it is not 
emphasized or included in MCAS. 

a 6. Other reason (please specify) 



E5. Would you change your emphasis on the 
content of thus item if your students did not have to 
take MCAS? 

a 1. 1 still would not teach it. 

° 2. 1 would give it slight emphasis 

a 3. 1 would give it substantial emphasis 
Go to item F. 



E6. For which of the following reasons do you teach 
the content of this test item? (Select as many as 
apply.) 

a 1. It is part of the curriculum for this grade 
and course 

a 2. It is not part of the curriculum for this 
grade and course, but I include it as review or to 
help students learn other material included in 
the curriculum. 

a 3. It helps students do well on MCAS. 

a 4. It helps students do well on other tests. 

a 5. It is preparation for more advanced 
mathematics courses 

E7. How much emphasis do you give to the content 
of this test item? 

a 1. It is taught in one lesson; not the primary 
focus of that lesson. 

a 2. It is the primary or sole focus of a single 
lesson. 

a 3. It is the primary or sole focus of two 
lessons. 

a 4. It is taught over an extended period of 
time (e.g., woven through an investigation). 

E8. Which of the following best characterizes the 
way you teach this content? 

a 1. 1 have presented this test item 

a 2. 1 present problems with similar content 
and similar format 

a 3. 1 present problems with similar content 
that do not resemble this test item 

D 4. 1 present this content in the context of 
other topics but do not present problems that 
focus primarily on this content 
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from question E 5 



▼ 

Go to Item F. 



E9. Do you make reference to MCAS when you 
teach the content of this item? 

□ 1. No 

□ 2. Yes; I tell students that this general 
content is likely to be included in MCAS. 

□ 3. Yes; I tell students that an item like this 
one is likely to be included in MCAS. 

□ 4. Yes; I tell students that this general 
content is not likely to be included in MCAS. 

a 5. Yes; I tell students that items like this are 
not likely to be included in MCAS. 



E10. Would your emphasis on the content of this 
item change if your students did not have to take 



MCAS? 



a 1. It would decrease substantially 
a 2. It would decrease slightly 
a 3. It would not change 
O 4. It would increase slightly 
□ 5. It would increase substantially 

Go to Item F. 



Item F . Use the balance scales below to answer the question below. 

Figure 1 




How many cylinders must be placed on the empty side of the second scale to make that scale balance? 

A. 5 

B. 2 

C. 3 

D. 4 



FI. Do you recognize this test item? 

□ 1. No 

° 2. Yes, from MCAS 

a 3. Yes, from another test 

Which test? 

° 4. Yes, from test-preparation materials, a textbook, or other instructional materials 

Which materials? 
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F2. What must a student know or be able to do in order to answer this item correctly? Please briefly note the skill 
or knowledge you consider most essential, (e.g., "must be able to do simple computations with exponents.") 



F3. Do you teach content of this test item in this class? 

a 1. No, I don't teach this content. Continue to question F4 . 

a 2. 1 teach this content. Skip to question FiL 

a 3. I teach similar content but not exactly this (e.g., "I teach area, but only with regular polygons"). 
Skip to question F6 . 

If you answered “no” to question F3, continue to question F4. Otherwise, skip to question F6. 



F4. Why do you not teach this content? 

a 1. It is not in the curriculum. 

° 2. It is taught in an earlier grade. 

° 3. It is taught in a later grade. 

a 4. It is in the curriculum, but it is not an 
important topic. 

a 5. It is in the curriculum but it is not 
emphasized or included in MCAS. 

a 6. Other reason (please specify) 



F5. Would you change your emphasis on the 
content of this item if your students did not have to 
take MCAS? 

3 1. 1 still would not teach it. 

3 2. I would give it slight emphasis 

3 3. 1 would give it substantial emphasis 

Go to Item G. 



▼ 



F6. For which of the following reasons do you teach 
the content of this test item? (Select as many as 
apply.) 

3 1. It is part of the curriculum for this grade 

and course 

3 2. It is not part of the curriculum for this 

grade and course, but I include it as review or to 
help students learn other material included in 
the curriculum. 

a 3. It helps students do well on MCAS. 

3 4. It helps students do well on other tests. 

a 5. It is preparation for more advanced 
mathematics courses 

F7. How much emphasis do you give to the content 
of this test item? 

3 1. It is taught in one lesson; not the primary 

focus of that lesson. 

3 2. It is the primary or sole focus of a single 

lesson. 

3 3. It is the primary or sole focus of two 

lessons. 

3 4. It is taught over an extended period of 

time (e.g., woven through an investigation). 

F8. Which of the following best characterizes the 
way you teach this content? 

3 1. 1 have presented this test item 

3 2. 1 present problems with similar content 

and similar format 

a 3. 1 present problems with similar content 
that do not resemble this test item 

3 4.1 present this content in the context of 

other topics but do not present problems that 
focus primarily on this content 
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from question F5 



Go to Item G . 



t 



F9. Do you make reference to MCAS when you 
teach the content of this item? 

□ 1. No 

□ 2. Yes; I tell students that this general 
content is likely to be included in MCAS 

□ 3. Yes; I tell students that an item like this 
one is likely to be included in MCAS. 

□ 4. Yes; I tell students that this general 
content is not likely to be included in MCAS. 

□ 5. Yes; I tell students that items like this are 
not likely to be included in MCAS. 

F10. Would your emphasis on the content of this 
item change if your students did not have to take 
MCAS? 



□ 


1. It would decrease substantially 


□ 


2. It would decrease slightly 


□ 


3. It would not change 


□ 


4. It would increase slightly 


□ 


5. It would increase substantially 


Go to 


Item G 



Item G . The pattern shown below is for a square prism. The lengths of the line segments in the pattern were 
chosen so that the pattern could be folded along the dotted lines into the prism shown. 



a. Make a sketch of a pattern for a triangular prism. Label each line segment with a 
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length that will make it possible to fold the pattern into the triangular prism. 



b. Make a sketch of a pattern for a cylinder. Label each line segment and 

diameter in your pattern with a length that will make it possible to create the cylinder from 
the pattern. 



Gl. Do you recognize this test item? 

□ 1. No 

D 2. Yes, from MCAS 

a 3. Yes, from another test 

Which test? 

a 4. Yes, from test-preparation materials, a textbook, or other instructional materials 



Which materials? 
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G2. What must a student know or be able to do in order to answer this item correctly? Please briefly note the 
skill or knowledge you consider most essential, (e.g., "must be able to do simple computations with exponents/') 



G3. Do you teach content of this test item in this class? 

G 1. No, I don't teach this content. Continue to question G4 . 

G 2. 1 teach this content. Skip to question G6. 

G 3. I teach similar content but not exactly this (e.g., "I teach area, but only with regular polygons"). 
Skip to question G6 . 

If you answered “no” to question G3, continue to question G4. Otherwise, skip to question G6. 



G4. Why do you not teach this content? 

G 1. It is not in the curriculum. 

G 2. It is taught in an earlier grade. 

G 3. It is taught in a later grade. 

G 4. It is in the curriculum, but it is not an 
important topic. 

G 5. It is in the curriculum but it is not 
emphasized or included in MCAS. 

G 6. Other reason (please specify) 



G5. Would you change your emphasis on the 
content of this item if your students did not have to 
take MCAS? 

G 1. 1 still would not teach it. 

G 2. 1 would give it slight emphasis 
G 3. 1 would give it substantial emphasis 
Go to Item H. 



▼ 



G6. For which of the following reasons do you teach 
the content of this test item? (Select as many as 
apply.) 

G 1. It is part of the curriculum for this grade 
and course 

G 2. It is not part of the curriculum for this 
grade and course, but I include it as review or to 
help students learn other material included in 
the curriculum. 

G 3. It helps students do well on MCAS. 

G 4. It helps students do well on other tests. 

G 5. It is preparation for more advanced 
mathematics courses 

G7. How much emphasis do you give to the content 
of this test item? 

G 1. It is taught in one lesson; not the primary 
focus of that lesson. 

G 2. It is the primary or sole focus of a single 
lesson. 

G 3. It is the primary or sole focus of two 
lessons. 

G 4. It is taught over an extended period of 
time (e.g., woven through an investigation). 
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from question G5 



▼ 



Go to Item H. 



G8. Which of the following best characterizes the 
way you teach this content? 

□ 1. 1 have presented this test item 

□ 2. 1 present problems with similar content 
and similar format 

□ 3.1 present problems with similar content 
that do not resemble this test item 

a 4.1 present this content in the context of 
other topics but do not present problems 

G9. Do you make reference to MCAS when you 
teach the content of this item? 

□ 1. No 

□ 2. Yes; I tell students that this general 
content is likely to be included in MCAS. 

□ 3. Yes; I tell students that an item like this 
one is likely to be included in MCAS. 

□ 4. Yes; I tell students that this general 
content is not likely to be included in MCAS. 

a 5. Yes; I tell students that items like this are 
not likely to be included in MCAS. 

G10. Would your emphasis on the content of this 
item change if your students did not have to take 
MCAS? 

a 1. It would decrease substantially 

□ 2. It would decrease slightly 

□ 3. It would not change 

a 4. It would increase slightly 

□ 5. It would increase substantially 
Go to Item H . 



TO NEXT PAGE 
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Item H . This question requires you to show your work and explain your reasoning. You may use drawings, 
words, and numbers in your explanation. Your answer should be clear enough so that another person could read 
it and understand your thinking. It is important that you show all you work. 



In a game, Carla and Maria are making subtraction problems using tiles numbered 1 to 5. 
The player whose subtraction problem gives the largest answer wins the game. 



Look at where each girl placed two of her tiles. 

Carla 





] 








- 


s 






Maria 











- 




1 




Who will win the game? 

Explain how you know this person will win. 



HI. Do you recognize this test item? 

□ 1. No 

a 2. Yes, from MCAS 
a 3. Yes, from another test 

Which test? 

a 4. Yes, from test-preparation materials, a textbook, or other instructional materials 

Which materials? 

H2. What must a student know or be able to do in order to answer this item correctly? Please briefly note the 
skill or knowledge you consider most essential, (e.g., "must be able to do simple computations with exponents.") 
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H3. Do you teach content of this test item in this class? 

□ 1. No, I don't teach this content. Continue to question H4 . 

a 2. 1 teach this content. Skip to question H£ ► 

□ 3. I teach similar content but not exactly this (e.g., "I teach area, but only with regular polygons"). 

Skip to question H6 . ^ 

If you answered “no” to question H3, continue to question H4. Otherwise, skip to question H6. 



^ H4. Why do you not teach this content? 

a 1. It is not in the curriculum. 

a 2. It is taught in an earlier grade. 

a 3. It is taught in a later grade. 

a 4. It is in the curriculum, but it is not an 
important topic. 

O 5. It is in the curriculum but it is not 
emphasized or included in MCAS. 

a 6. Other reason (please specify) 



H5. Would you change your emphasis on the 
content of this item if your students did not have to 
take MCAS? 

a 1. 1 still would not teach it. 
a 2. 1 would give it slight emphasis 
a 3. 1 would give it substantial emphasis 
Go to Item I. 



* 



T 

H6. For which of the following reasons do you teach 
the content of this test item? (Select as many as 
apply.) 

□ 1. It is part of the curriculum for this grade 
and course 

□ 2. It is not part of the curriculum for this 
grade and course, but I include it as review or to 
help students learn other material included in 
the curriculum. 

□ 3. It helps students do well on MCAS. 

□ 4. It helps students do well on other tests. 

□ 5. It is preparation for more advanced 
mathematics courses 

H7. How much emphasis do you give to the content 
of this test item? 

□ 1. It is taught in one lesson; not the primary 
focus of that lesson. 

□ 2. It is the primary or sole focus of a single 
lesson. 

□ 3. It is the primary or sole focus of two 
lessons. 

□ 4. It is taught over an extended period of 
time (e.g., woven through an investigation). 

H8. Which of the following best characterizes the 
way you teach this content? 

□ 1. 1 have presented this test item 

□ 2. 1 present problems with similar content 
and similar format 

□ 3. 1 present problems with similar content 
that do not resemble this test item 

□ 4. 1 present this content in the context of 
other topics but do not present problems that 
focus primarily on this content 



73 



75 



from question H5 



H9. Do you make reference to MCAS when you 
teach the content of this item? 



t 



Go to Item I. 



□ 1. No 

a 2. Yes; I tell students that this general 
content is likely to be included in MCAS. 

a 3. Yes; I tell students that an item like this 
one is likely to be included in MCAS. 

a 4. Yes; I tell students that this general 
content is not likely to be included in MCAS. 

a 5. Yes; I tell students that items like this are 
not likely to be included in MCAS. 



H10. Would your emphasis on the content of this 
item change if your students did not have to take 



MCAS? 



a 1. It would decrease substantially 
□ 2. It would decrease slightly 

a 3. It would not change 
a 4. It would increase slightly 
a 5. It would increase substantially 
Go to Item I . 



Item I . If each of the counting numbers from 1 through 10 is multiplied by 13, how many of the 
resulting numbers will be even? 

A) One 

B) Four 

C) Five 

D) Six 

E) Ten 



II. Do you recognize this test item? 

□ 1. No 

a 2 . Yes, from MCAS 
° 3. Yes, from another test 

Which test? 

4 . Yes, from test-preparation materials, a textbook, or other instructional materials 

Which materials? 
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12. What must a student know or be able to do in order to answer this item correctly? Please briefly note the skill 
or knowledge you consider most essential. (e.g v "must be able to do simple computations with exponents.") 



13. Do you teach content of this test item in this class? 

a 1. No, I don't teach this content. Continue to question 14 . 

a 2. 1 teach this content. Skip to question 16, 

□ 3. I teach similar content but not exactly this (e.g., "I teach area, but only with regular polygons"). 

Skip to question 16 . 

If you answered “no” to question 13, continue to question 14. Otherwise, skip to question 16. 



^ 14 . Why do you not teach this content? 

O 1. It is not in the curriculum. 

a 2. It is taught in an earlier grade. 

a 3. It is taught in a later grade. 

a 4. It is in the curriculum, but it is not an 
important topic. 

a 5. It is in the curriculum but it is not 
emphasized or included in MCAS. 

a 6. Other reason (please specify) 



15. Would you change your emphasis on the content 
of this item if your students did not have to take 
MCAS? 

a 1. 1 still would not teach it. 
a 2. 1 would give it slight emphasis 
a 3.1 would give it substantial emphasis 
Go to Item T. 



16. For which of the following reasons do you teach 
the content of this test item? (Select as many as 
apply.) 

□ 1. It is part of the curriculum for this grade 
and course 

□ 2. It is not part of the curriculum for this 
grade and course, but I include it as review or to 
help students learn other material included in 
the curriculum. 

□ 3. It helps students do well on MCAS. 

□ 4. It helps students do well on other tests. 

□ 5. It is preparation for more advanced 
mathematics courses 

17. How much emphasis do you give to the content 
of this test item? 

O 1. It is taught in one lesson; not the primary 
focus of that lesson. 

□ 2. It is the primary or sole focus of a single 
lesson. 

□ 3. It is taught in two lessons. 

□ 4. It is taught in three or more lessons. 

18. Which of the following best characterizes the 
way you teach this content? 

□ 1. 1 have presented this test item 

□ 2. 1 present problems with similar content 
and similar format 

□ 3. 1 present problems with similar content 
that do not resemble this test item 

O 4. 1 present this content in the context of 
other topics but do not present problems that 
focus primarily on this content 
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from question 15 



19. Do you make reference to MCAS when you 
teach the content of this item? 

□ 1. No 

a 2. Yes; I tell students that this general 
content is likely to be included in MCAS. 

O 3. Yes; I tell students that an item like this 
one is likely to be included in MCAS. 

a 4. Yes; I tell students that this general 
content is not likely to be included in MCAS. 

O 5. Yes; I tell students that items like this are 
not likely to be included in MCAS. 

110. Would your emphasis on the content of this 
item change if your students did not have to take 
MCAS? 

a 1. It would decrease substantially 
a 2. It would decrease slightly 
a 3. It would not change 
a 4. It would increase slightly 
a 5. It would increase substantially 
Go to Item J . ' G o to Item I . 



Item J . Erin is writing a science fiction story. She has invented a money system for her planet that 
uses four coins that she drew and name like this: 




She has challenged her classmates to determine the relationships among the values of the 



CLUE 1: 

3 ©s are worth the same as 1 A and 1 ©. 



CLUE 2: j —* i— | i— | 

Kay bought a game costing \2/ IS IS. 

She gave the clerk 2 ^**)s. Her change was 1 0. 



CLUE 3: 

Kay and Max have the same amount of money. 
Kay has 1 0 , 4 ©s, and 1®. 

Max has 3 0s and 6 As. 



coins from the following clues. 
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a. Use Clue 1 above to find how many (c)s equal 1 /^\ . Use words or pictures to explain 
your reasoning. 

b. Use Clue 2 to find how many §s equal 1 <H> . Use words or pictures to explain your 
reasoning. 

c. Use Clue 3 and your answers to parts a and b to find how many equal 1 @ . 

Use words or pictures to explain your reasoning. 

d. Erin told her classmates that 1 is worth 25$ in U.S. money. What is the value in U.S. 
money of each of the following? 

• i ^ 

• i @ 

• 1 < 2 > 

e. Use Clue 1 above to find how many ©s equal 1 . Use words or pictures to explain 

your reasoning. 

f. Use Clue 2 to find how many [r]s equal 1 {h} . Use words or pictures to explain your 
reasoning. 

g. Use Clue 3 and your answers to parts a and b to find how many /t\ s equal 1 [r] . 

Use words or pictures to explain your reasoning. 

h. Erin told her classmates that 1 (c) is worth 25$ in U.S. money. What is the value in U.S. 
money of each of the following? 

• i ^ 

• i @ 

• 1 <H> 

TO NEXT PACE 
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Jl. Do you recognize this test item? 

□ 1. No 

a 2. Yes, from MCAS 
a 3. Yes, from another test 

Which test? 

a 4. Yes, from test-preparation materials, a textbook, or other instructional materials 

Which materials? 

J2. What must a student know or be able to do in order to answer this item correctly? Please briefly note the skill 
or knowledge you consider most essential, (e.g., "must be able to do simple computations with exponents/ 7 ) 



J3. Do you teach content of this test item in this class? 

a 1. No, I don't teach this content. Continue to question 14 . 

a 2. 1 teach this content. Skip to question 16 I 

a 3. I teach similar content but not exactly this (e.g., "I teach area, but only with regular polygons"). 
Skip to question J6 . 

If you answered “no” to question J3, continue to question J4. Otherwise, skip to question J6. 



J4. Why do you not teach this content? 

a 1. It is not in the curriculum. 

a 2. It is taught in an earlier grade. 

a 3. It is taught in a later grade. 

a 4. It is in the curriculum, but it is not an 
important topic. 

a 5. It is in the curriculum but it is not 
emphasized or included in MCAS. 

a 6. Other reason (please specify) 



J5. Would you change your emphasis on the content 
of this item if your students did not have to take 
MCAS? 

a 1. 1 still would not teach it. 
a 2. 1 would give it slight emphasis 
a 3. 1 would give it substantial emphasis 
Go to next page. 



J6. For which of the following reasons do you teach 
the content of this test item? (Select as many as 
apply.) 

a 1. It is part of the curriculum for this grade 
and course 

a 2. It is not part of the curriculum for this 
grade and course, but I include it as review or to 
help students learn other material included in 
the curriculum. 

a 3. It helps students do well on MCAS. 

a 4. It helps students do well on other tests. 

a 5. It is preparation for more advanced 
mathematics courses 



▼ 
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from question J5 



Go to next pag e. 



▼ 



J7. How much emphasis do you give to the content 
of this test item? 

□ 1. It is taught in one lesson; not the primary 
focus of that lesson. 

□ 2. It is the primary or sole focus of a single 
lesson. 

□ 3. It is the primary or sole focus of two 
lessons. 

□ 4. It is taught over an extended period of 
time (e.g., woven through an investigation). 

J8. Which of the following best characterizes the 
way you teach this content? 

O 1. 1 have presented this test item 

□ 2. 1 present problems with similar content 
and similar format 

□ 3. 1 present problems with similar content 
that do not resemble this test item 

O 4. 1 present this content in the context of 
other topics but do not present problems that 
focus primarily on this content 

J9. Do you make reference to MCAS when you 
teach the content of this item? 

□ 1. No 

a 2. Yes; I tell students that this general 
content is likely to be included in MCAS. 

□ 3. Yes; I tell students that an item like this 
one is likely to be included in MCAS. 

o 4. Yes; I tell students that this general 
content is not likely to be included in MCAS. 

a 5. Yes; I tell students that items like this are 
not likely to be included in MCAS. 

J10. Would your emphasis on the content of this 
item change if your students did not have to take 
MCAS? 

O 1. It would decrease substantially 
a 2. It would decrease slightly 
a 3. It would not change 
O 4. It would increase slightly 
a 5. It would increase substantially 
Go to next page . 
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Questions about content Areas 

In this section, we will ask you a series of questions about several mathematics content areas. Each area is 
described in terms of a student proficiency. 

NOTE: When we use the term "content/' it refers to the knowledge AND the skills needed to answer a given 
test item. 

A. Use proportional thinking to solve problems (including rates , scaling, and similarity) 

Al. Do you teach this content in the target class? 

a 1. No, I don't teach this content. Go to question A2 . 

° 2. I teach this content. Skip to question AjL ^ 

a 3. 1 teach similar content but not exactly this. Skip to question A4 ^ 

If you answered “no” to question Al. continue to question A2. Otherwise* skip to question A4. 



A2. Why do you not teach this content in the target 
class? 

a 1 . It is not in the curriculum. 

° 2. It is taught in an earlier grade. 

a 3. It is taught in a later grade. 

a 4. It is in the curriculum, but it is not an 
important topic. 

a 5. It is in the curriculum but it is not 
emphasized or included in MCAS. 

a 6. Other reason (please specify) 



A3. How much emphasis would you give this 
content if your students did not take the MCAS 
assessment? 

a 1. I still would not teach it 

a 2. I would teach it but would not give it 
much emphasis. 

a 3. I would give it substantial emphasis. 
Go to next content-area listing . 



A4. For which of the following Reasons do you teach 
this content ? (Select as many^as apply.) 

O 1. It is part of the curriculum for this grade 
and course 

a 2. It is not part of the curriculum for this 
grade and course, but I include it as review or to 
help students learn other material included in 
the curriculum. 

O 3. It helps students do well on MCAS. 

O 4. It helps students do well on other tests. 

O 5. It is preparation for more advanced 
mathematics courses 

A5. Would your emphasis on this content change if 
your students did not have to take MCAS? 

O 1. It would decrease substantially 

O 2. It would decrease slightly 

O 3. It would not change 

O 4. It would increase slightly 

O 5. It would increase substantially 

Go to next content-area listing . 



▼ 
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B . Solve problems involving percentages 



Bl. Do you teach this content in the target class? 

a 1. No, I don't teach this content. Go to question B2 . 

a 2. I teach this content. Skip to question B4 ? ► 

a 3. 1 teach similar content but not exactly this. Skip to q uestion B4 ^ 

If you answered “no” to question Bl, continue to question B2 Otherwise, skip to question B4. 



w B2. Why do you not teach this content in the target 
class? 

a 1. It is not in the curriculum. 

a 2. It is taught in an earlier grade. 

a 3. It is taught in a later grade. 

a 4. It is in the curriculum, but it is not an 
important topic. 

a 5. It is in the curriculum but it is not 
emphasized or included in MCAS. 

6. Other reason (please specify) 



B3. How much emphasis would you give this 
content if your students did not take the MCAS 
assessment? 

a 1. I still would not teach it 

a 2. I would teach it but would not give it 
much emphasis. 

a 3. I would give it substantial emphasis. 



B4. For which of the following reasons do you 
teach this content ? (Select as many as apply.) 

□ 1 . It is part of the curriculum for this grade 
and course 

a 2. It is not part of the curriculum for this 
grade and course, but I include it as review or to 
help students learn other material included in 
the curriculum. 

a 3. It helps students do well on MCAS. 

□ 4. It helps students do well on other tests. 

□ 5. It is preparation for more advanced 
mathematics courses 



B5. Would your emphasis on this content change if 
your students did not have to take MCAS? 

□ 1. It would decrease substantially 

□ 2. It would decrease slightly 
a 3. It would not change 

a 4. It would increase slightly 
a 5. It would increase substantially 



Go to next content-area listing. 



Go to the next content area . 



C. Solve problems involving perimeter and area (such as triangles , quadrilaterals , other polygons, 
circles , combined forms) 

Cl. Do you teach this content in the target class? 

a 1. No, I don't teach this content. Go to question C2 . 

a 2. I teach this content. Skip to question C4, ^ 

a 3. 1 teach similar content but not exactly this. Skip to question C4 

If you answered “no” to question Cl, continue to question C2. Otherwise, skip to question C4. 

C2. Why do you not teach this content in the target class? 
a 1. It is not in the curriculum. 



' a 2. It is taught in an earlier grade. 

a 3. It is taught in a later grade. 

a 4. It is in the curriculum, but it is not an 
important topic. 

a 5. It is in the curriculum but it is not 
emphasized or included in MCAS. 

a 6. Other reason (please specify) 



C3. How much emphasis would you give this 
content if your students did not take the MCAS 
assessment? 

a 1. I still would not teach it 

° 2. I would teach it but would not give it 

much emphasis. 

a 3. I would give it substantial emphasis. 
Go to next content-area listing. 



C4. For which of the following reasons do you teach 
this content ? (Select as many as apply.) 

a 1. It is part of the curriculum for this grade 
and course 

a 2. It is not part of the curriculum for this 
grade and course, but I include it as review or to 
help students learn other material included in 
the curriculum. 

° 3. It helps students do well on MCAS. 

a 4. It helps students do well on other tests. 

a 5. It is preparation for more advanced 
mathematics courses 

C5. Would your emphasis on this content change if 
your students did not have to take MCAS? 

a 1. It would decrease substantially 

a 2. It would decrease slightly 

a 3. It would not change 

a 4. It would increase slightly 

a 5. It would increase substantially 

Go to the next content area . 
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D. Use transformations (translations, rotations, reflections, dilations, symmetry) 



Dl. Do you teach this content in the target class? 



□ 1. No, I don't teach this content. Go to question D2 . 



a 2. I teach this content. Skip to question D4 

□ 3. 1 teach similar content but not exactly this. Skip to question D4- 



If von answered ‘hio” to question Dl. continue to question D2. Otherwise, skip to question D4. 



D2. Why do you not teach this content in the target class? 
O 1. It is not in the curriculum. 
a 2. It is taught in an earlier grade. 



D4. For which of the following reasons do you teach 
this content ? (Select as many as apply.) 

□ 1. It is part of the curriculum for this grade 

and course 



□ 3. It is taught in a later grade. 

a 4. It is in the curriculum, but it is not an 
important topic. 

□ 5. It is in the curriculum but it is not 
emphasized or included in MCAS. 

6. Other reason (please specify) 



□ 2. It is not part of the curriculum for this 
grade and course, but I include it as review or to 
help students learn other material included in 
the curriculum. 

□ 3. It helps students do well on MCAS. 

□ 4. It helps students do well on other tests. 

□ 5. It is preparation for more advanced 
mathematics courses 



D3. How much emphasis would you give this 
content if your students did not take the MCAS 
assessment? 

a 1. I still would not teach it 

a 2.1 would teach it but would not give it 
much emphasis. 

a 3. I would give it substantial emphasis. 
Go to next content-area listing. 



D5. Would your emphasis on this content change if 
your students did not have to take MCAS? 

□ 1. It would decrease substantially 
a 2. It would decrease slightly 

□ 3. It would not change 

□ 4. It would increase slightly 

a 5. It would increase substantially 



Go to the next content area . 
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E . Use the properties of geometric figures to determine the measurement of angles 

El. Do you teach this content in the target class? 

— a 1. No, I don't teach this content. Go to question E2 . 

° 2. I teach this content. Skip to question E4 ^ 

a 3. 1 teach similar content but not exactly this. Skip to question E4 ^ 

If you answered to question EL continue to question E2. Otherwise, skip to question E4. 



E2. Why do you not teach this content in the target class? 

° 1. It is not in the curriculum. 

a 2. It is taught in an earlier grade. 

a 3. It is taught in a later grade. 

° 4. It is in the curriculum, but it is not an 

important topic. 

° 5. It is in the curriculum but it is not 

emphasized or included in MCAS. 

6. Other reason (please specify) 



E3. How much emphasis would you give this 
content if your students did not take the MCAS 
assessment? 

a 1. I still would not teach it 

a 2. I would teach it but would not give it 
much emphasis. 

a 3. I would give it substantial emphasis. 
Go to next content-area listing. 



E4. For which of the following reasons do you teach 
this content ? (Select as many as apply.) 

□ 1. It is review of material taught in an earlier 
1. It is part of the curriculum for this grade and 
course 

O 2. It is not part of the curriculum for this 
grade and course, but I include it as review or to 
help students learn other material included in 
the curriculum. 

□ 3. It helps students do well on MCAS. 

a 4. It helps students do well on other tests. 

o 5. It is preparation for more advanced 
mathematics courses 

E5. Would your emphasis on this content change if 
your students did not have to take MCAS? 

O 1. It would decrease substantially 

a 2. It would decrease slightly 

a 3. It would not change 

a 4. It would increase slightly 

o 5. It would increase substantially 



Go to the next content area . 
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F. Extend a pattern or functional relationship 

FI. Do you teach this content in the target class? 

— □ 1. No, I don't teach this content. Go to question F2 . 

O 2. I teach this content. Skip to question F4 ► 

□ 3. 1 teach similar content but not exactly this. Skip to question FA ^ 

If you answered “no” to question FI, continue to question F2. Otherwise* skip to question F4. 

F2. Why do you not teach this content in the target class? 

O 1. It is not in the curriculum. 



a 2. It is taught in an earlier grade. 

a 3. It is taught in a later grade. 

a 4. It is in the curriculum, but it is not an 
important topic. 

a 5. It is in the curriculum but it is not 
emphasized or included in MCAS. 

6. Other reason (please specify) 



F3. How much emphasis would you give this 
content if your students did not take the MCAS 
assessment? 

a 1. I still would not teach it 

a 2. I would teach it but would not give it 
much emphasis. 

a 3. I would give it substantial emphasis. 
Go to next content-area listing. 



F4. For which of the following reasons do you teach 
this content ? (Select as many as apply.) 

□ 1. It is part of the curriculum for this grade 
and course 

□ 2. It is not part of the curriculum for this 
grade and course, but I include it as review or to 
help students learn other material included in 
the curriculum. 

□ 3. It helps students do well on MCAS. 

□ 4. It helps students do well on other tests. 

□ 5. It is preparation for more advanced 
mathematics courses 

F5. Would your emphasis on this content change if 
your students did not have to take MCAS? 

□ 1. It would decrease substantially 
O 2. It would decrease slightly 

□ 3. It would not change 

a 4. It would increase slightly 

□ 5. It would increase substantially 

Go to the next content area . 
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G. Solve systems of equations and inequalities using appropriate methods 



Gl. Do you teach this content in the target class? 

a 1. No, I don't teach this content. Go to question G2 . 

a 2. I teach this content. Skip to question G4 ^ 

a 3.1 teach similar content but not exactly this. Skip to question G4 ^ 

If you answered “no” to question Gl, continue to question G2. Otherwise, skip to question G4. 



G2. Why do you not teach this content in the target class? 

a 1. It is not in the curriculum. 

° 2. It is taught in an earlier grade. 

° 3. It is taught in a later grade. 

a 4. It is in the curriculum, but it is not an 
important topic. 

a 5. It is in the curriculum but it is not 
emphasized or included in MCAS. 

6. Other reason (please specify) 



G3. How much emphasis would you give this 
content if your students did not take the MCAS 
assessment? 

a 1. I still would not teach it 

a 2. I would teach it but would not give it 
much emphasis. 

a 3.1 would give it substantial emphasis. 



G4. For which of the following reasons do you teach 
this content ? (Select as many as apply.) 

a 1. It is part of the curriculum for this grade 
and course 

a 2. It is not part of the curriculum for this 
grade and course, but I include it as review or to 
help students learn other material included in 
the curriculum. 

° 3. It helps students do well on MCAS. 

° 4. It helps students do well on other tests. 

a 5. It is preparation for more advanced 
mathematics courses 



G5. Would your emphasis on this content change 
if your students did not have to take MCAS? 

a 1. It would decrease substantially 

a 2. It would decrease slightly 

a 3. It would not change 

D 4. It would increase slightly 

a 5. It would increase substantially 
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